Technical Lead - Training Runtime, Data(set) Movement

The role

This Technical Lead will own datasets throughout the training infrastructure. You will design and build a unified dataset read platform and set the direction for how training jobs read data.

What you will do

• Design and build a unified dataset read platform for multiple training frameworks.
• Define dataset APIs, storage-format expectations, registration, versioning, and migration paths.
• Build reliability into the read path including stateful iteration, caching, fast restart, and recovery.
• Build terminal and web-based visualizers to inspect data late in the pipeline.
• Write and review production code in core data loading, service, caching, and reliability paths.
• Partner with teams working on training frameworks, reinforcement learning, multimodal models, storage, runtime, and cluster infrastructure.

Required qualifications

• Have built or owned dataset, data loading, storage, or distributed training infrastructure at large scale.
• Understand the failure modes of large distributed training jobs.
• Have experience with stateful iterators, checkpoint/restart semantics, caching, remote services, or high-throughput storage reads.
• Are comfortable working across Python and lower-level systems code.

Nice to have

• Rust or C++ experience is useful but not required
• Have worked with multimodal, video, reinforcement learning, or pretraining data pipelines

Before you apply

• Background checks for applicants will be administered in accordance with applicable law

About the company

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity.