Senior HPC Engineer - Fleet Engineering

The role

You will build and operate monitoring and alerting for cluster health. You will remotely deploy and configure large-scale HPC clusters for AI workloads using automation.

What you will do

• Build and operate monitoring and alerting for cluster health
• Remotely deploy and configure large-scale HPC clusters for AI workloads using automation
• Automate cluster lifecycle including operating systems, firmware, drivers, and networking
• Create runbooks and automated remediations for common cluster failure modes
• Troubleshoot and resolve cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, switching, and power
• Participate in on-call rotations and lead incident response for cluster-level problems
• Contribute to and maintain Standard Operating Procedures

Required qualifications

• 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role
• Strong understanding of modern AI infrastructure, from GPU architectures to hardware performance optimization
• Strong understanding of Linux-based systems in a distributed environment
• Experienced configuring and troubleshooting InfiniBand (IB), RoCE, CLOS fabrics, 100GbE, Ethernet/switching, GPU-direct, and NCCL environments
• Solid understanding of Python and Go
• Experience working with SWE teams to improve internal tooling
• Experience with monitoring and alerting tools (e.g., Prometheus, Grafana, Clickhouse)
• Proficiency in automation and configuration management tools (e.g., Ansible, Terraform)
• Excellent problem-solving and troubleshooting skills
• Innate attention to detail

Nice to have

• Experience with machine learning / deep learning frameworks (PyTorch, TensorFlow) and benchmarking tools (DeepSpeed, MLPerf)
• Knowledge of containerization and orchestration technologies (e.g., Docker, Kubernetes)
• Experience building and/or operating HPC resources
• Depth in the NVIDIA hardware and firmware ecosystem
• Experience with data center power and thermal design
• Background in chaos engineering or similar reliability testing methodologies
• Understanding of compliance frameworks (SOC 2, ISO 27001, etc.)

Benefits

• Generous cash & equity compensation
• Health, dental, and vision coverage for you and your dependents
• Wellness and commuter stipends for select roles
• 401k Plan with 2% company match (USA employees)
• Flexible paid time off plan

About the company

Lambda is a leader in AI cloud infrastructure serving tens of thousands of customers. Founded in 2012, the company has 500+ employees and is growing fast.