Senior Staff Software Engineer, DC Infrastructure

The role

This person will develop software to manage a fleet of GPU servers and data centers. They will build diagnostic, observability, automation, and repair tooling for high-performance GPU compute clusters.

What you will do

• Developing and implementing deep-level diagnostics and troubleshooting of hardware faults within GPU racks and high-density compute systems.
• Developing troubleshooting and automation tooling for GPU platforms including NVIDIA A100, H200, GB200, B200 and AMD 350X / 355X.
• Developing automation and AI agents for executing component-level diagnosis and remediation for failed or degraded hardware.
• Develop innovative tooling and AI agents for managing the critical environment in conjunction with data center operations.
• Developing tooling for post-repair validation and testing tools such as burn-in, Pytorch, and NVIDIA NCCL.
• Own the deployment, monitoring, and operational support of developed tooling.
• Developing automation and operational tooling for facilities management power as well as direct liquid cooling hardware systems.

Required qualifications

• Software engineering experience.
• The ability to identify a problem, rapidly develop a scalable solution and ship it.
• Ability to lean in and assist team members working on critical or complex technical initiatives.
• Ability to set the technical direction for a specific project and execute.
• Expertise in distributed systems, reliability, and cloud platforms (Kubernetes, IaC, GCP etc.).
• Strength in at least one programming language - Go, Python, Java, Rust.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.
• Ability to work independently and within a team

Nice to have

• Experience with Temporal and Kubernetes.
• Experience working directly with hardware vendors.
• Background in large-scale GPU fleet operations or hyperscale data center environments.

Benefits

• Industry competitive pay
• Restricted Stock Units
• Health insurance package options that include HDHP and PPO, vision, and dental for you and your dependents
• Employer contributions to HSA accounts
• Paid Parental Leave
• Paid life insurance, short-term and long-term disability
• Teladoc
• 401(k) with a 100% match up to 4% of salary
• Generous paid time off and holiday schedule
• Cell phone reimbursement
• Tuition reimbursement
• Subscription to the Calm app

About the company

Crusoe is a vertically integrated AI infrastructure company that owns and operates each layer of the stack to power AI workloads. They use an energy-first approach to provide AI hardware and infrastructure as a service.