Senior Infrastructure Engineer

The role

You will build tools, automation, and data platforms to support the hardware lifecycle. This includes developing frameworks for debugging, validation, and monitoring high-performance compute systems.

What you will do

• Design, build, and maintain software tools and frameworks for hardware development and sustaining lifecycle.
• Develop automated test, diagnostics, and health-monitoring frameworks for server fleets.
• Build tooling, monitoring and logging systems to help hardware engineers triage issues.
• Create automated validation and characterization workflows for GPU, CPU, and high-performance compute platforms.
• Build and operate data pipelines, dashboards, and analytics for hardware failure trends and reliability metrics.
• Support E2E integration and solution testing by developing test harnesses and CI/CD workflows.
• Evaluate and introduce new tools, testing methods, and automation approaches.
• Provide data-driven insights from tooling and telemetry to influence hardware roadmap and reliability strategy.

Required qualifications

• 5+ years of professional software engineering experience, ideally building internal tools, infrastructure software, provisioning, test automation, or systems software.
• Strong proficiency in Python and at least one systems language (Go, C/C++, or Rust), plus Shell scripting.
• Experience designing and implementing automation and test frameworks, including workflow engines, for hardware or large-scale infrastructure.
• Experience with BMC/Redfish/IPMI, firmware update automation, or out-of-band management.
• Experience with fleet management, provisioning, or large-scale automation tools (Kubernetes, Ansible, Terraform, etc.).
• Experience building data pipelines, logging, telemetry, or observability tooling to analyze system behavior at scale.
• Experience working across hardware, firmware, and software teams in multidisciplinary environments.
• Strong analytical and problem-solving skills with a data-driven approach.
• Excellent communication and collaboration skills.
• Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent experience.

Nice to have

• Experience building tooling for GPU clusters, including GPU-to-GPU communication diagnostics (NCCL tests, NVLink monitoring, etc.)
• Familiarity with PCIe, InfiniBand, and NVMe/storage concepts and tooling (e.g., lspci, ibstat, nvme-cli, perftest, fio)
• Working knowledge of Linux systems, including kernel interfaces, drivers, device tooling, and low-level system diagnostics
• Familiarity with NVIDIA tooling such as DCGM, NVML, and fabric management software
• Experience supporting both ARM and x86 server architectures
• Background in factory test, manufacturing test software, or hardware qualification workflows

About the company

Crusoe is a vertically integrated AI infrastructure company that owns and operates each layer of the stack to power AI workloads. They use an energy-first approach to build AI infrastructure.