AI Compute Engineer

The role

You will lead the architecture, qualification, and operational readiness of GPU infrastructure across scaling data centers. You will define technical models connecting GPU servers, firmware, OS, NVIDIA software, and high-performance fabrics.

What you will do

• Define the architecture and set lifecycle standards for NVIDIA HGX platforms.
• Lead qualification and burn-in strategy for new clusters and GPU platforms.
• Diagnose health and performance by interpreting NVIDIA diagnostics and telemetry.
• Validate with real workloads including dense and MoE parallelism strategies.
• Integrate across domains including Network, Storage, Platform/DevOps, and Facilities.
• Own operational readiness and acceptance by defining health and readiness gates.
• Guide automation teams in building reproducible workflows.
• Translate customer requirements into validation strategies.

Required qualifications

• Strong hands-on experience with NVIDIA GPU infrastructure, including HGX or comparable large GPU server platforms.
• Demonstrated experience designing, executing, or leading burn-in and qualification across multi-node GPU clusters on H100, H200, B200, B300, or a comparable platform.
• Experience analyzing fleet-level qualification results, performance distributions, and hardware/software failures.
• Strong Linux systems administration and troubleshooting.
• Experience with NVIDIA drivers, CUDA, NCCL, DCGM, and GPU platform tooling.
• Experience diagnosing GPU, NVLink, NVSwitch, PCIe, thermal, power, or firmware problems.
• Experience validating intra-node and multi-node GPU communication, plus a working understanding of RDMA, GPUDirect RDMA, and high-performance AI fabrics.
• Experience deploying, provisioning, or operationalizing GPU clusters.
• Experience defining firmware, BIOS, OS, and software baselines, as well as health, quarantine, readiness, and return-to-service criteria.
• Experience interpreting distributed training or inference workload behavior.
• Ability to troubleshoot performance across GPU, host, network, storage, and software boundaries.
• Ability to define requirements for automation teams and interpret the outputs of automated validation systems.

Nice to have

• NVIDIA B200 or B300 infrastructure.
• ConnectX adapters, NVIDIA Quantum, InfiniBand, UFM, RoCE, or comparable networking.
• Large-scale NCCL and distributed workload qualification.
• PyTorch, NeMo, Megatron, vLLM, or TensorRT-LLM.
• GPU schedulers such as Slurm, Kubernetes, Run:ai, Base Command, or equivalent.
• Validating shared storage and distributed checkpointing on high-performance platforms such as WEKA, VAST, or equivalent.
• Infrastructure automation, observability, telemetry, or fleet-management platforms.
• Formal customer acceptance testing.
• Operating GPU infrastructure across multiple data centers.
• Python, Bash, or equivalent scripting for troubleshooting infrastructure workflows.

About the company

breakmark is building a next-generation AI infrastructure platform that delivers high-performance bare-metal GPU compute for large-scale AI training and inference.