Post-Training Research Engineer

The role

You will build in-house tooling to support training a wide spectrum of model architectures using various techniques efficiently and at scale. This involves working across the stack with systems-level concepts, PyTorch distributed tensor computation, and GPU kernels.

What you will do

• Build in-house tooling to support model training
• Train a wide spectrum of different model architectures with a variety of techniques efficiently and at scale
• Work across the stack including systems-level concepts like Kubernetes, cgroups, storage systems, and networking topologies
• Work with PyTorch distributed tensor computation and GPU kernels
• Work with researchers to derive specifications and execute
• Profile and improve the performance of a distributed GPU program
• Perform roofline analysis on a transformer training setup

Required qualifications

• Deep understanding of modern ML techniques and tools for training transformers
• Advanced experience in a tensor/array computation library like PyTorch, TensorFlow, Jax, or similar
• Detailed understanding of transformer training parallelism strategies like data parallelism, sharded data parallelism, tensor parallelism, pipeline parallelism, context parallelism
• Experience and knowledge to profile and improve the performance of a distributed GPU program in PyTorch or a similar library
• Ability to perform roofline analysis on a transformer training setup
• Willingness to dive into messy problems, work with researchers, derive specifications by asking important questions, and execute
• Familiarity with HPC and distributed computing platforms like Slurm, Ray, Kubernetes, and Dask
• Familiarity with cluster networking technology like Infiniband, RoCE, GPUDirect
• Solid fundamentals in operating systems concepts like processes, files, kernel drivers, containerization, and networking protocols
• A sense of creativity and willingness to ask difficult questions about our approach, assumptions, and tooling choices

Benefits

• Competitive compensation, including meaningful equity
• 100% coverage of medical, dental, and vision insurance for employee and dependents (U.S. only)
• Flexible PTO policy including company wide Winter Break
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k) (U.S. only)
• Exposure to a variety of ML startups

About the company

Baseten powers mission-critical inference for AI companies by uniting applied AI research, flexible infrastructure, and seamless developer tooling. The company recently raised a $1.5B Series F.