Software Engineer, Model Runtime

The role

You will build the model runtime within the inference engine to execute frontier models on custom silicon. You will work across model architecture, distributed systems, compilers, kernels, and silicon to optimize throughput, latency, and reliability.

What you will do

• Design and implement the LLM inference runtime for frontier models running on custom silicon.
• Build scheduling, continuous batching, memory management, KV-cache management, and execution orchestration.
• Develop distributed execution strategies across chips, hosts, and racks.
• Optimize end-to-end latency, throughput, memory efficiency, and hardware utilization.
• Partner with kernel, compiler, architecture, and silicon teams to co-design interfaces.
• Enable new model features, execution patterns, numerical formats, and hardware capabilities.
• Create profiling, observability, benchmarking, and performance-modeling tools.
• Debug complex correctness, performance, and reliability issues.
• Turn workload insights into requirements for future silicon and system architecture.

Required qualifications

• Strong systems programming experience in C++, Rust, Python, or comparable performance-oriented environments.
• Built or optimized runtimes, distributed systems, compilers, kernels, model-serving infrastructure, or adjacent systems software.
• Understand modern LLM inference, including prefill and decode behavior, batching, KV-cache tradeoffs, and model parallelism.
• Can reason quantitatively about latency, throughput, compute intensity, memory bandwidth, communication, and utilization.
• Can profile and debug performance across multiple layers of a hardware-software stack.
• Can design clean abstractions while retaining low-level control for specialized hardware.
• Work effectively across model, systems, compiler, kernel, and hardware teams.
• Care about production quality, including correctness, observability, reliability, maintainability, and graceful behavior at scale.

About the company

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. The Hardware organization develops AI-native silicon and system-level solutions for advanced AI workloads.