Senior Software Engineer, ML API – AI Inference

The role

You will build and evolve the ML API layer for disaggregated AI inference systems. You will own features across inference APIs, model integration layers, and request-routing services.

What you will do

• Design, implement, test, and operate features for chat completions, text generation, and multimodal inputs.
• Write design documents and coordinate implementation with engineers and partner teams.
• Implement request and response semantics across inference backends.
• Integrate foundation models, tokenizers, and sampling methods into the serving platform.
• Extend inference services with frameworks like vLLM, PyTorch, and Hugging Face.
• Implement routing and request lifecycle management for disaggregated inference.
• Diagnose and improve time to first token, streaming latency, and throughput.
• Build conformance tests, model-validation suites, and performance benchmarks.
• Improve service metrics, tracing, logging, alerts, and diagnostic tools.
• Provide design and code reviews and mentor less-experienced engineers.

Required qualifications

• 5+ years of software engineering experience, including independent ownership of production services, distributed systems, or substantial software components.
• Strong programming ability in Python and proficiency in Go, C++, Rust, or another language used to build production services.
• Experience designing and delivering features that span multiple components.
• Experience building and evolving production APIs.
• Familiarity with REST, gRPC, or streaming protocols.
• Experience integrating software across service, framework, runtime, or infrastructure boundaries.
• Experience operating services on Linux using containers, Kubernetes or comparable orchestration systems, and CI/CD.
• Ability to diagnose correctness, reliability, and performance issues across distributed components.
• Strong communication and collaboration skills.
• Ability to turn ambiguous requirements into maintainable software.
• Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.

Nice to have

• Experience building or operating production ML or LLM inference APIs, including OpenAI-compatible interfaces.
• Experience extending or contributing to vLLM, SGLang, PyTorch, Hugging Face Transformers, Triton, TensorRT-LLM, or another ML systems project.
• Understanding of tokenization, chat templates, generation configuration, logits processing, stopping criteria, tool calling, structured generation, or constrained decoding.
• Experience with disaggregated prefill/decode architectures, KV-cache transfer, prefix caching, chunked prefill, or inference request scheduling.
• Experience building API conformance, model-quality, numerical-comparison, determinism, or performance-regression test systems.
• Experience with multi-model or multi-tenant serving, including routing, admission control, quotas, rate limiting, or capacity-aware scheduling.
• Experience developing performance-sensitive or highly concurrent services in Go, C++, Rust, or a similar language.
• Familiarity with GPU inference, AMD Instinct accelerators, ROCm, CUDA, collective communication, or multi-node model serving.
• Familiarity with reduced-precision inference and quantization formats such as BF16, FP8, FP4, INT8, or INT4.
• Experience building SDKs, developer tools, configuration systems, or self-service ML workflows, or collaborating directly with customers and open-source communities.

Benefits

• Build a breakthrough AI platform beyond the constraints of the GPU.
• Publish and open source cutting-edge AI research.
• Work on one of the fastest AI supercomputers in the world.
• Job stability with startup vitality.
• Simple, non-corporate work culture that respects individual beliefs.

About the company

Cerebras Systems builds the world's largest AI chip. The architecture delivers industry-leading training and inference speeds.