Site Reliability Engineer - Fleet Command & Control

The role

You will build the command-and-control layer for a fleet of frontier inference systems. This includes designing alerting, automated recovery, and fleet data to keep clusters serving.

What you will do

• Define healthy inference system signals and automated actions
• Set common alerting standards and define SLOs
• Suppress alert noise and correlate related alerts
• Automate hardware health recovery with guardrails
• Route events to the correct engineers and technicians
• Shape fleet data and set tracing standards using OpenTelemetry
• Run incident management including postmortems and on-call rotations
• Review service designs for failure modes and graceful degradation
• Acknowledge and triage user impact during incidents
• Debug production incidents using logs, metrics, traces, and core dumps

Required qualifications

• 2+ years in site reliability, production engineering or infrastructure for large-scale production systems
• Defined alerting standards using Prometheus, Grafana or similar
• Real on-call experience for services with external users
• Track record of debugging production incidents in systems you did not write
• Experience with application and scripting languages like python, golang, rust, etc.
• Hands-on with Linux, containers, Kubernetes and infrastructure as code (Terraform, Ansible or similar)
• Fluency with Linux debugging tools (perf, strace, gdb, dmesg, journalctl, lspci, tcpdump)
• Comfort with hardware in the loop
• Sound judgment about when automation should act on its own and when it should page a person
• Clear writing and ability to get five or more teams to agree on shared standards

Nice to have

• Operated GPU, accelerator or HPC fleets
• LLM inference serving: prefill and decode, KV cache, batching, and latency metrics
• Network debugging from switch to edge
• Out-of-band management (BMC, Redfish, IPMI) and fleet-wide firmware rollouts
• Data center telemetry: power distribution, liquid cooling and CDUs, DCIM
• PagerDuty or similar incident tooling
• Working with colocation providers and remote-hands technicians
• Founding or scaling an SRE function at a startup

Benefits

• Medical, dental, and vision packages with generous premium coverage
• $500 per month credit for waiving medical benefits
• Housing subsidy of $2,500 per month for those living within walking distance of the office
• Relocation support for those moving to San Jose (Santana Row)
• Various wellness benefits covering fitness, mental health, and more
• Daily lunch and dinner in our office
• Unlimited compute budget subject to ROI justification

About the company

Etched is building hardware for frontier intelligence by co-designing chips, racks, software, and manufacturing. They are a fully in-person team in San Jose.