Performance Engineer, Inference

Bengaluru. Posted 3 months ago

The role

You will own the production serving path for large distributed models end to end. You will integrate model and kernel artifacts into a multi-node, multi-tenant stack and build and train speculators.

What you will do

Own the production serving path for large distributed models end to end
Operate and extend a distributed-serving stack
Integrate artifacts from the model and kernel teams into a running multi-node, multi-tenant stack
Build and train speculators including draft models and distillation
Produce and defend latency and throughput numbers
Work with architecture co-design, kernels team, model team, and SRE
Maintain on-call ownership of an inference SLO

Required qualifications

5+ years in ML systems
2+ years on inference serving at production scale
Experience serving 100B+ parameter models in production across multi-node tensor, pipeline, or expert parallelism
Source-level fluency in one of SGLang, vLLM, Dynamo, or TensorRT-LLM
Reading-level familiarity with the other three of SGLang, vLLM, Dynamo, or TensorRT-LLM
Experience with distributed serving including disaggregated prefill-decode stack, distributed KV/cache transfer, and cross-node routing and scheduling
Speculative decoding as a build-and-train competency
Deep understanding of KV cache internals
Working command of TP / PP / EP, NCCL primitives, and how they interact with the scheduler
Multi-tenant serving experience
Profiling fluency with Nsight Systems, framework tracing, and py-spy / perf
C++ and CUDA at a read-and-modify level

Nice to have

Upstream contributions to SGLang, vLLM, Dynamo, llm-d, TensorRT-LLM, or LMDeploy
Direct production experience with Dynamo or llm-d at scale
Having operated a forked runtime in production
Published or shipped speculator work
MoE serving at scale
Long-context (128K+)
Multi-model serving
Indic and multilingual workloads

About the company

Sarvam serves multiple model families including small and large LLMs, Mixture-of-Experts, Indic ASR & TTS, streaming models and multimodal models across a multi-node, multi-tenant fleet of Hoppers and Blackwells.