Member of Technical Staff - Distributed Training Engineer

The role

This role focuses on runtime, performance, and reliability for distributed training systems. You will design and build infrastructure to power next-generation Liquid Foundation Models.

What you will do

• Design and build core systems that make large training runs fast and reliable
• Build scalable distributed training infrastructure for GPU clusters
• Implement and tune parallelism/sharding strategies for evolving architectures
• Optimize distributed efficiency including topology-aware collectives, comm/compute overlap, and straggler mitigation
• Build data loading systems that eliminate I/O bottlenecks for multimodal datasets
• Develop checkpointing mechanisms balancing memory constraints with recovery needs
• Create monitoring, profiling, and debugging tools for training stability and performance

Required qualifications

• Hands-on experience building distributed training infrastructure (PyTorch Distributed DDP/FSDP, DeepSpeed ZeRO, Megatron-LM TP/PP)
• Experience diagnosing performance bottlenecks and failure modes (profiling, NCCL/collectives issues, hangs, OOMs, stragglers)
• Understanding of hardware accelerators and networking topologies
• Experience optimizing data pipelines for ML workloads

Nice to have

• MoE (Mixture of Experts) training experience
• Large-scale distributed training (100+ GPUs)
• Open-source contributions to training infrastructure projects

Benefits

• Greenfield challenges
• High ownership from day one
• Competitive base salary with equity
• 100% medical, dental, and vision premiums for employees and dependents
• 401(k) matching up to 4% of base pay
• Unlimited PTO
• Company-wide Refill Days throughout the year

About the company

Spun out of MIT CSAIL, liquid-ai builds general-purpose AI systems that run efficiently across deployment targets. They partner with enterprises across consumer electronics, automotive, life sciences, and financial services.