Training Performance Engineer

The role

You will drive efficiency improvements across the distributed training stack. You will analyze large-scale training runs, identify utilization gaps, and design optimizations for throughput and uptime.

What you will do

• Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.
• Optimize GPU utilization and throughput for large-scale distributed model training.
• Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.
• Implement model graph transforms to improve end to end throughput.
• Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.
• Partner with researchers to ensure new model architectures scale efficiently during pre-training.
• Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs.

Required qualifications

• Strong programming skills in Python and C++
• Experience running distributed training jobs on multi-GPU systems or HPC clusters
• Exposure to frameworks like PyTorch, JAX, or TensorFlow
• Understanding of how large-scale training loops are built

Nice to have

• Rust or CUDA
• Familiarity with NCCL, MPI, or UCX communication libraries
• Experience with large-scale data loading and checkpointing systems
• Prior work on training runtime, distributed scheduling, or ML compiler optimization

Benefits

• Relocation assistance

Before you apply

• Background checks for applicants will be administered in accordance with applicable law

About the company

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity.