ML Infrastructure Engineer - New Grad

The role

You will write production Python for the inference serving stack and its vLLM integration. You will implement features like speculative decoding and prompt caching on the RDU.

What you will do

• Implement inference features on the RDU such as speculative decoding, constrained decoding, tool calling, prompt caching, and long-context inference
• Adapt vLLM and related serving frameworks to the RDU architecture
• Build and maintain endpoints in the public inference API
• Write accuracy verification and regression tests for inference features
• Debug latency, throughput, and reliability issues in production services

Required qualifications

• B.S. in Computer Science, Electrical Engineering, or related field
• Strong programming skills in Python
• Understanding of algorithms, data structures, concurrency, and systems design
• Coursework or project experience in distributed systems or machine learning
• Experience with LLM serving frameworks such as vLLM, SGLang, or TensorRT-LLM
• Experience with inference techniques such as speculative decoding, KV caching, or continuous batching
• Experience with PyTorch

Nice to have

• M.S. or Ph.D. in Computer Science, Electrical Engineering, or related field
• Experience building or maintaining APIs or backend services
• Internship or research experience in ML systems or backend infrastructure
• Experience with disaggregated systems - prefill/decode
• Open source project ownership or contributions

Benefits

• Base salary plus equity
• 95% premium coverage for employee medical insurance
• 77% premium coverage for dependents
• Health Savings Account (HSA) with employer contribution
• Dental, Vision, Short/Long term Disability, Basic Life, Voluntary Life, and AD&D insurance plans
• Flexible Spending Account (FSA) options like Health Care, Limited Purpose, and Dependent Care
• Headspace subscription
• Gympass+ membership
• One Medical membership
• Counseling services with an Employee Assistance Program

Before you apply

• You must submit an application form for each position for which you believe you are qualified

About the company

SambaNova is a leader in next-generation AI infrastructure, delivering a full-stack inference platform for customers worldwide.