Software Engineer, Inference

The role

You will define and build inference architecture across self-hosted models and third-party providers. You will work on systems for serving, routing, capacity management, and optimizing for latency, reliability, and cost.

What you will do

• Partner with frontier labs and providers for capacity, training and inference infrastructure.
• Shape Sierra's inference architecture including serving and proxy layers.
• Build systems for routing, failover, capacity management, and quota.
• Build and operate self-hosted inference on GPU infrastructure.
• Optimize inference performance using techniques like speculative decoding.
• Work across a hybrid inference stack with Sierra-managed and third-party platforms.
• Push the serving stack forward by tuning engines and infrastructure.
• Contribute to infrastructure that enables post-training.

Required qualifications

• Deep systems thinking and strong distributed systems fundamentals.
• Experience designing, building, and operating large-scale production systems.
• Strong judgment around tradeoffs involving latency, reliability, capacity, and cost.
• Experience taking ownership of complex infrastructure from architecture through production operation.
• Excitement about applying systems expertise to AI infrastructure.

Nice to have

• Experience with ML infrastructure, MLOps, or production inference systems.
• Experience serving LLMs or other large models at scale.
• Experience operating self-hosted inference and GPU infrastructure.
• Familiarity with inference frameworks such as vLLM or SGLang.
• Experience with post-training infrastructure or inference-performance optimization.

Benefits

• Flexible (unlimited) paid time off
• Medical, dental, and vision benefits for you and your family
• Life insurance and disability benefits
• Retirement plan dependent on country of employment
• Parental leave
• Fertility and family building benefits
• Lunch, as well as delicious snacks and coffee when working onsite
• Discretionary benefit stipend
• Free alphorn lessons
• Equity plans for eligible full-time employees

About the company

Sierra is a platform for customer-facing AI agents. It is an in-person company based in San Francisco.