Product Manager, Inference Platform

The role

You will work with inference engineering and research teams to turn advances in model serving and performance into customer-ready capabilities. You will define benchmarks, prioritize optimizations based on customer workloads, and help customers adopt performance features in production.

What you will do

• Work with inference engineering and research teams to bring new performance capabilities to customers.
• Translate advances in batching, caching, quantization, speculative decoding, and serving architectures into useful product features.
• Define benchmarks and evaluations that show improvements in latency, throughput, quality, and cost.
• Understand customer workloads and help prioritize the optimizations with the greatest impact.
• Make performance features easier to configure, evaluate, and adopt in production.
• Help customers understand tradeoffs across hardware, model configurations, autoscaling, and routing.
• Partner with engineering on release readiness, compatibility, and regression testing.

Required qualifications

• Experience building and shipping AI/ML products or infrastructure, especially where performance, scalability, or production behavior mattered.
• A technical background in software engineering, ML engineering, research, or equivalent hands-on experience.
• Strong understanding of model serving architectures, GPU economics, and latency, throughput, quality, and cost tradeoffs.
• Experience working closely with researchers or infrastructure engineers to bring technically complex capabilities into production.
• A rigorous approach to benchmarking, experimentation, and performance measurement.
• Strong product judgment and the ability to translate deep technical work into clear customer value.

Nice to have

• Hands-on experience with PyTorch, vLLM, TensorRT-LLM, SGLang, NVIDIA Dynamo, or similar technologies.
• Experience with open-source inference projects, model optimization, or ML systems research.

Benefits

• Competitive compensation, including meaningful equity.
• (U.S. only) 100% coverage of medical, dental, and vision insurance for employee and dependents.
• Flexible PTO policy, including company-wide Winter Break (offices closed from Christmas Eve to New Year's Day).
• Paid parental leave.
• Fertility and family-building stipend through Carrot.
• (U.S. only) Company-facilitated 401(k).
• Exposure to a variety of ML startups, with learning and networking opportunities.

Before you apply

• The company is an Equal Opportunity Employer and will consider qualified applicants with criminal histories in line with applicable law, including the San Francisco Fair Chance Ordinance where applica

About the company

Baseten powers mission-critical inference for AI companies such as Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. The company recently raised a $1.5B Series F led by Altimeter Capital, Conviction Partners, and Spark Capital.