Crusoe
Senior Network Production Engineer, Operations
San Francisco, CA - US. Posted a month ago
The role
This person will support production reliability across global network infrastructure including edge, backbone, data center fabric, and GPU cluster interconnects. They will focus on incident response, root cause analysis, and automation-driven operational execution for hyperscale AI infrastructure.
What you will do
- Support uptime across global edge, backbone, data center, and GPU cluster network.
- Write and maintain Python-based tooling to reduce toil and automate remediation workflows.
- Participate in high-severity network events including detection, triage, and mitigation.
- Perform root cause analysis using existing and newly built tooling.
- Build automation on top of the monitoring stack to reduce manual triage.
- Maintain and improve runbooks, escalation playbooks, and SOPs.
- Help build and maintain automated dashboards and alerting for reliability metrics.
- Work with Architecture and SRE teams to execute operational excellence initiatives.
Required qualifications
- 5+ years of production network engineering experience with a focus on operations, incident response, and reliability in large-scale or internet-scale environments.
- Strong Python and scripting proficiency.
- Hands-on experience with observability and monitoring tools including streaming telemetry, SNMP, NetFlow/sFlow, Grafana, Prometheus, and ThousandEyes.
- Ability to script against monitoring APIs.
- Experience operating RDMA/RoCE lossless fabrics for GPU or HPC workloads, including familiarity with PFC, ECN, and DCQCN tuning.
- Solid hands-on knowledge of BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, and TCP/IP in production data center environments.
- Proficiency with Arista (EOS) and Juniper (Junos) platforms in leaf-spine CLOS architectures.
- Comfort operating large device fleets across multi-region environments with on-call responsibility.
- Bachelor's degree in Computer Science, Electrical Engineering, or a related field, or equivalent practical experience.
Nice to have
- Experience with NVIDIA/Mellanox networking platforms in GPU cluster environments.
- Familiarity with Kentik or Arbor for traffic analysis and DDoS visibility.
- Exposure to contributing to SLIs and SLOs alongside SRE or product teams.
- Exposure to operating 10K+ device fleets across hyperscale or cloud environments.
Benefits
- Competitive compensation
- Restricted Stock Units
- Paid time off & paid holidays
- Comprehensive health, dental & vision insurance
- Employer contributions to HSA account
- Paid parental leave
- Paid life insurance, short-term and long-term disability
- Professional development & tuition reimbursement
- Mental health & wellness support
- Commuter benefits (parking & transit)
- Cell phone stipend
- 401(k) Retirement plan with company match up to 4% of salary
About the company
Crusoe is a vertically integrated AI infrastructure company that owns and operates each layer of the stack to power AI workloads. They use an energy-first approach to solve power bottlenecks for AI compute.