AI Infrastructure Engineer

The role

You will own the accelerator platform end to end, managing clusters and schedulers across multiple cloud and on-prem environments. You will write production code to make compute fast, fault-tolerant, and fully utilized.

What you will do

• Design, deploy and operate GPU clusters across AWS, GCP, and other hyperscalers/neoclouds/on-premise data centers
• Build scheduling for distributed jobs, bin-packing, and quotas for multi-tenant teams
• Run distributed training infrastructure on PyTorch Distributed, Ray, or Megatron
• Deploy and tune high-throughput LLM serving with vLLM or similar
• Build benchmarking and traffic-replay tools to measure performance baselines
• Build health checks and automated remediation for hardware, network and software layers
• Instrument fleet utilization with DCGM, Prometheus and Grafana
• Ingest utilization, occupancy and billing data into a warehouse for capacity planning
• Drive autoscaling, rightsizing and idle-capacity recovery programs
• Write clear runbooks and documentation

Required qualifications

• 5+ years building and operating production distributed systems or ML infrastructure
• Write production-quality Python plus Go or C++
• Fluent in shell scripting for system administration
• Deep hands-on Kubernetes and container experience
• Working knowledge of Slurm or another HPC resource manager
• Have run GPU workloads at scale, including scheduling, orchestration and troubleshooting
• Hands-on experience with at least one distributed training or serving stack: PyTorch Distributed, Ray, vLLM or Megatron-LM
• Deep experience with at least one major cloud provider
• Know observability tools like DataDog, Prometheus, PromQL and Grafana
• Comfortable on-call for critical systems
• Own projects end to end and gather requirements with little direction

Nice to have

• LLM inference optimization: KV cache systems, disaggregated serving, quantization, speculative decoding
• High-performance networking: RDMA, InfiniBand, NCCL, NVLink
• GPU profiling with Nsight or similar, and CUDA or Triton kernel work
• Accelerators beyond NVIDIA: AMD MI-series, TPU, Trainium
• Capacity planning, billing normalization or cost attribution across providers
• Automated fault diagnosis, log mining or AIOps at fleet scale
• Configuration management (Ansible, Puppet) and secure or air-gapped deployments

Benefits

• Bi-annual performance bonus structure
• Generous equity grant vested over 4 years
• Up to $15k Relocation bonus
• $10K housing bonus (if you live within 0.5 miles of our office)
• $1.5K monthly stipend for meals
• Free Equinox membership
• $200 monthly laundry reimbursement
• $200 monthly personal wellness reimbursement
• Health, Dental, Vision insurance

About the company

Mercor builds infrastructure that enables frontier AI labs and enterprises to train, evaluate, and improve AI on real-world professional work. The company is profitable and recently raised a $750 million Series D at a $20 billion valuation.