Staff Software Engineer, Inference API

The role

You will build and evolve the ML API layer for disaggregated AI inference systems. This involves working across inference APIs, model integration layers, request-routing services, and the Cerebras inference platform.

What you will do

• Build production ML inference APIs for chat completions, text generation, and multimodal inputs.
• Deliver a unified serving experience across heterogeneous inference backends.
• Integrate emerging foundation models, tokenizers, and sampling methods into the serving platform.
• Maintain API compatibility and establish versioning and deprecation practices.
• Integrate custom inference services with vLLM, PyTorch, Hugging Face, and AMD ROCm stack.
• Support disaggregated inference by building control and data paths for GPU prefill and Cerebras decode.
• Optimize streaming behavior, latency, throughput, and communication between components.
• Build validation systems for tokenization, sampling, and model upgrades.
• Define service indicators and build logging, tracing, metrics, and diagnostic tooling.
• Develop testing infrastructure including conformance tests and performance benchmarks.

Required qualifications

• 5+ years of software engineering experience, including substantial individual-contributor ownership of production software or distributed systems.
• Strong programming ability in Python and Go.
• Experience developing performance-sensitive or highly concurrent services in C++, Rust, or a similar systems language.
• Experience building stable APIs with clear validation, error handling, observability, compatibility, and versioning practices.
• Experience integrating software across service, framework, runtime, and infrastructure boundaries.
• Experience designing or maintaining OpenAI-compatible, gRPC, REST, or streaming inference APIs.
• Experience with Linux, containers, Kubernetes or comparable orchestration systems, CI/CD, and operating latency-sensitive services in production.
• Ability to diagnose correctness, reliability, and performance issues across multiple components of a distributed serving system.
• Strong communication and cross-functional execution skills.
• Bachelor's degree in computer science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.

Nice to have

• Experience modifying or contributing to vLLM, SGLang, PyTorch, Hugging Face Transformers, Triton, TensorRT-LLM, or another open-source ML systems project.
• Experience creating API conformance, model-quality, numerical-comparison, determinism, or performance-regression test systems
• Experience building SDKs, developer tools, model registries, configuration systems, or self-service ML platforms.
• Experience with multi-model or multi-tenant inference platforms, including routing, admission control, fairness, quotas, rate limiting, and capacity-aware scheduling
• Understanding of model-specific tokenization, chat templates, generation configuration, logits processing, stopping criteria, tool calling, structured generation, and constrained decoding.
• Experience with disaggregated prefill/decode architectures, KV-cache transfer, prefix caching, chunked prefill, memory-aware admission control, or request scheduling.
• Experience designing, building, or operating production APIs and services for machine learning, large language models, or other data-intensive applications.
• Familiarity with reduced-precision inference and quantization formats such as BF16, FP8, FP4, INT8, or INT4.

Benefits

• Build a breakthrough AI platform beyond the constraints of the GPU.
• Publish and open source cutting-edge AI research.
• Work on one of the fastest AI supercomputers in the world.
• Job stability with startup vitality.
• Simple, non-corporate work culture that respects individual beliefs.

About the company

Cerebras Systems builds the world's largest AI chip, 56 times larger than GPUs. The architecture delivers industry-leading training and inference speeds.