Tech Lead Manager, GPU Cluster Infrastructure

The role

You will lead the infrastructure sub-team and act as an engineer, spending at least half your time on technical work. You will own the platform's technical direction, roadmap, and the hiring and growth of the team.

What you will do

• Set the platform's technical direction and own its roadmap
• Own architecture and the scheduling and storage design
• Debug failures such as node health, GPU and fabric faults, and multi-node job hangs
• Hire and grow a small team of senior engineers
• Set security direction for a shared cluster
• Decide how the team operates, including on-call, incident response, and observability
• Act as the escalation point for research teams and providers

Required qualifications

• Led engineers as a manager, tech lead, or project lead
• 5+ years in systems or infrastructure engineering on production Linux
• Experience running GPU, HPC, or large-scale batch platforms
• Owned systems from design through operation
• Depth in at least one of scheduling, storage, networking, security, or GPU systems
• Breadth to review designs in the rest
• Run production Kubernetes for GPU workloads with a batch layer on top
• Owned infrastructure as code and observability for a production fleet
• Strong programmer in at least one language such as Python, Go, Rust, or C++
• Write clearly for engineers, researchers, and providers

Nice to have

• Distributed training infrastructure: multi-node PyTorch and NCCL debugging, the NVIDIA node stack, InfiniBand or RoCE fabrics, topology-aware placement
• Distributed storage: VAST, Weka, Lustre, Ceph, or object storage at scale; checkpoint I/O
• Cluster security: admission control, RBAC, node and container hardening, sandboxed runtimes, and isolating autonomous agents
• Scheduler internals: Kubernetes scheduler plugins or custom controllers, gang scheduling, fair-share and quota
• Multi-provider platforms: scheduling and storage across clusters at different providers
• Greenfield team-building: hired senior engineers and stood up on-call, incident, and review practices

Benefits

• Health Insurance - 94% of Insurance premium paid by Organization commencing within 1 month after your start date
• Retirement - 401(k) plan with up to 2% match
• 25 days Paid Time Off per year, accrued weekly
• Up to 10 days of paid sick leave per year
• Paid Bereavement, Family, Medical and Pregnancy Disability Leave
• Work computer and stipend provided for eligible employees
• Catered lunches and dinners on workdays at our Berkeley office
• Work-related travel and equipment expenses paid

Before you apply

• Sponsor visas for in-person employees
• One hour programming assessment
• Short screening call
• One hour code review interview
• 90 minute systems design and incident response interview

About the company

FAR.AI is a non-profit AI research institute working to ensure advanced AI is safe and beneficial for everyone. Their work includes facilitating breakthrough AI safety research and conducting pre-deployment testing for frontier developers.