Senior ML Engineer | Kimchi (Inference Stack Optimization)

The role

You will optimize the inference stack to improve throughput, latency, cost per token, and margins. You will work on SGLang and vLLM engines, weight loaders, KV caches, and distributed inference topologies.

What you will do

• Push throughput via continuous batching, chunked prefill, speculative decoding, and scheduler tuning
• Cut latency by profiling and fixing compute, memory, scheduling, storage, or network bottlenecks
• Optimize KV cache through hierarchical and paged attention, prefix caching, and quantization
• Reduce cold starts using weight loaders, JIT and compile caches, and shared-filesystem read paths
• Quantize weights and activations without regressing quality
• Scale across nodes using distributed inference topologies and prefill/decode disaggregation
• Set technical direction by deciding what to benchmark, adopt, or build

Required qualifications

• 5+ years building real ML systems with depth in inference or training infrastructure
• Production-grade Python
• Experience running vLLM, SGLang, or TensorRT-LLM
• Ability to measure quantization regressions
• Knowledge of multi-GPU and multi-node setups including tensor and expert parallelism
• Knowledge of page caches, O_DIRECT, and network filesystems under terabyte-scale reads
• Ability to instrument and benchmark performance

Nice to have

• Worked on LLM routing specifically
• Calibration, abstention, or conformal methods in production decision systems
• Ran online A/B loops with drift monitoring on a model that touches real money

Benefits

• Competitive salary
• Flexible, remote-first global environment
• Equity options
• Fast-paced workflow
• 10% of work time for personal projects or self-improvement
• Learning budget for professional and personal development
• Annual hackathon
• Team-building budget and company events
• Equipment budget
• Extra days off

Before you apply

• Cast AI does not provide any form of visa sponsorship/work permit
• A background check may be conducted at the final stage of recruitment through Checkr

About the company

Cast AI is a Miami-headquartered, globally distributed team across 34 countries. It hit unicorn status in January 2026 with a $50B+ backed investment.