Software Engineer, Hardware Health

The role

You will build critical infrastructure to keep large compute clusters healthy and operational at scale. You will define health signals, build automated remediation systems, and investigate hardware failures across GPUs, CPUs, and networking.

What you will do

• Define and maintain health signals across GPUs, CPUs, networking, and platform infrastructure.
• Build and evolve health checks that detect, remediate, and verify failures at scale.
• Ensure critical health checks execute with minimal latency.
• Investigate hardware failures and system-level issues across large-scale compute environments.
• Own node lifecycle workflows including drain, quarantine, repair, RMA, and return-to-service processes.
• Build automation and tooling for global cluster management.
• Partner with workload, reliability, and provider teams to integrate health signals.

Required qualifications

• 7+ years of industry experience in software or infrastructure engineering.
• Strong proficiency with Python and shell scripting.
• Experience building large-scale distributed systems or infrastructure platforms.
• Comfort digging into noisy operational data using SQL, PromQL, or similar tooling.
• Experience building reproducible analyses and operational tooling.
• Strong systems debugging and operational instincts with an ownership mindset.

Nice to have

• Experience with low-level hardware systems and Linux tooling (e.g. PCIe, InfiniBand, RoCE, networking, power management, kernel performance tuning, FW/SW debugging).
• Experience operating or debugging large-scale GPU or accelerator clusters.
• Expertise in network operations, observability, or systems telemetry.
• Experience with automated remediation systems or fleet lifecycle management.
• Experience improving reliability, utilization, or workload uptime in distributed compute environments.

Before you apply

• Background checks for applicants will be administered in accordance with applicable law.

About the company

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity.