Engineering Manager, Inference Performance

The role

You will lead a team of inference performance engineers working on the inference engine and runtime. You will manage people, set technical direction, and drive the productionization of inference techniques on GPUs.

What you will do

• Lead, mentor and grow a team of inference performance engineers through regular 1:1s, clear feedback, career development and performance reviews.
• Hire top GPU and inference engineering talent and build a strong, collaborative team culture.
• Own the technical roadmap and execution for runtime performance work.
• Review designs, guide profiling and optimization efforts.
• Drive the productionization of inference techniques such as quantization, speculative decoding, KV-cache reuse, chunked prefill and custom scheduling.
• Turn performance wins into measurable outcomes like tokens per GPU-hour, utilization, latency and cost.
• Help the team bring up and tune new model architectures on new hardware.
• Partner with Infrastructure, Inference Platform, Kernels, Model APIs and customer-facing teams.
• Set high standards for engineering quality, benchmarking, operational excellence and incident response.

Required qualifications

• Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or a related field.
• Experience managing engineers, including hiring, mentoring, giving feedback and running performance reviews.
• Experience leading or closely supporting GPU optimization teams in training, inference or recommendation systems.
• Strong technical depth in GPU workloads, with a solid understanding of GPU architecture and performance tradeoffs.
• Familiarity with ML libraries such as PyTorch, TensorRT or TensorRT-LLM.
• A track record of driving roadmaps and shipping complex technical projects with a team.
• Clear written and verbal communication, including the ability to align stakeholders across teams.

Nice to have

• Familiarity with inference engines such as vLLM, SGLang or TensorRT-LLM.
• Experience with LLM optimization techniques (e.g., quantization, speculative decoding, continuous batching) in production.
• Experience with GPU kernels (CUDA, Triton, CUTLASS, or similar).
• Experience scaling a team through rapid growth at a startup.
• A background as a hands-on performance or systems engineer before moving into management.

Benefits

• Competitive compensation, including meaningful equity
• 100% coverage of medical, dental, and vision insurance for employee and dependents (U.S. only)
• Flexible PTO policy including company wide Winter Break
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k) (U.S. only)
• Exposure to a variety of ML startups

About the company

Baseten powers mission-critical inference for AI companies by uniting applied AI research, flexible infrastructure, and seamless developer tooling. The company recently raised a $1.5B Series F.