Senior Site Reliability Engineer

The role

You will design, operate, and debug large-scale GPU infrastructure used for distributed training and inference. You will work directly with customers to optimize and troubleshoot workloads on multi-provider, multi-region GPU compute clusters.

What you will do

• Design and evolve multi-provider, multi-region GPU compute clusters.
• Serve as the primary technical point of contact for customers running large-scale training workloads.
• Define SLOs and error budgets for GPU infrastructure.
• Own capacity planning across heterogeneous GPU fleets.
• Ensure the health and performance of high-speed interconnects like InfiniBand, RoCE, and NVLink.
• Build visibility into GPU utilization, memory pressure, interconnect throughput, and hardware health.
• Build production-grade automation for cluster provisioning, health checks, and lifecycle management.
• Lead incident response for complex, multi-layer failures.
• Drive blameless postmortems and systemic fixes.

Required qualifications

• Hands-on experience operating large-scale GPU clusters (NVIDIA A100/H100/B200 or equivalent).
• Production experience with InfiniBand, RoCE, or NVLink fabrics.
• Working knowledge of NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, FSDP, or similar.
• Expert-level Linux knowledge including kernel tuning and driver management.
• Strong experience running Kubernetes in production with GPU workloads.
• Strong engineering skills in Python, Go, or Bash.
• Infrastructure-as-Code proficiency (Terraform, Helm, Ansible, or equivalent).
• Hands-on experience building monitoring and alerting for GPU infrastructure using GPU-specific telemetry.
• Proven track record leading incident response for complex distributed systems.

Nice to have

• Experience with high-performance parallel file systems (VAST, Weka, Lustre, GPFS).
• Experience profiling and optimizing distributed training performance.
• Experience involved in physical cluster design.
• Experience leading or mentoring a team of infrastructure engineers.

About the company

Andromeda is a market and infrastructure platform to buy, sell, and operate compute. The company aims to enable fragmented compute to flow through one platform to deliver reliable capacity to model builders, research labs, and inference providers.