Senior Site Reliability Engineer - Storage

The role

The Senior Site Reliability Engineer - Storage will own the reliability, performance, and capacity health of the production storage fleet. This role involves building monitoring, automating incident response, and designing self-healing automation for data platform services.

What you will do

• Own the reliability, performance, and capacity health of Lambda's production storage fleet.
• Build and maintain monitoring, dashboards, and alerting for storage performance, capacity, and hardware failures.
• Investigate and resolve storage-related incidents using telemetry, logs, and performance profiling.
• Automate ticketing, escalation, and incident-response workflows.
• Design and maintain self-healing automation for common failure modes.
• Implement CI/CD pipelines for storage automation and tooling.
• Partner with Storage, Fleet Orchestration, and Release Engineering to automate software-defined storage deployment.
• Work with hardware and networking teams to diagnose low-level I/O and network issues.
• Participate in an on-call rotation supporting the storage fleet.

Required qualifications

• 5+ years of experience operating Linux systems in production or HPC environments.
• Hands-on storage experience at scale on scale-out or software-defined platforms (e.g., CEPH, Lustre, GPFS, or similar).
• Hands-on experience operating Software-Defined Storage (SDS) platforms at scale, including integrating with management and data-plane APIs.
• Strong incident-response instincts.
• Working experience with monitoring and logging platforms such as Prometheus, Grafana, Alertmanager, Datadog, or SumoLogic.
• Working experience with Kubernetes (GitOps tooling such as ArgoCD, Helm/Kustomize) and hands-on troubleshooting.
• Working experience with CI/CD tooling (GitHub Actions, Jenkins, BuildKite), containerization (Docker/Podman), and systems programming in Python or Go.
• Working experience with Infrastructure as Code (Terraform, Ansible).
• Solid understanding of core storage protocols across file (NFS, SMB), object (S3), block (NVMe-oF/TCP), and structured (vector DB, SQL) storage.

Nice to have

• Experience with cutting-edge software-defined storage solutions such as VAST or Weka.
• Enterprise storage expertise in solutions such as NetApp, Dell PowerScale, GPFS, or Lustre.
• Experience writing or operating Kubernetes CSI drivers.
• Experience with SR-IOV and virtualization (KVM/QEMU).
• Experience with GPUDirect Storage, RDMA, InfiniBand, or RoCE networking.
• Familiarity with NIC-level diagnostics (ethtool, mlxlink) and fleet-wide operations tooling (clush or similar).
• Contributions to open-source storage projects.

Benefits

• Generous cash & equity compensation
• Health, dental, and vision coverage for you and your dependents
• Wellness and commuter stipends for select roles
• 401k Plan with 2% company match (USA employees)
• Flexible paid time off plan

About the company

Lambda is a leader in AI cloud infrastructure serving tens of thousands of customers. Founded in 2012, the company has 500+ employees and is growing fast.