Shieldai
Staff, Platform Engineer AI Compute (6034)
San Diego, California. Posted 11 days ago
The role
This person will build, maintain and automate the shared GPU compute platform that machine-learning engineers use to train and evaluate models. The work covers Kubernetes, scheduling, quotas, diagnostics and capacity planning.
What you will do
- Build and maintain the software and configuration that let developers run GPU workloads on Kubernetes.
- Design quotas, scheduling priorities, permissions and isolation so multiple teams can share compute predictably.
- Automate deployment and upgrades of GPU services, including Run:ai, with infrastructure as code and tools such as Terraform and Helm.
- Investigate why jobs wait, fail or run slowly, and fix recurring scheduling, configuration, storage and networking problems.
- Build diagnostics that connect queue times, GPU utilization, memory requirements and training throughput to the developer's experience.
- Benchmark representative workloads and recommend when to add GPU capacity, how much to add and which hardware fits the workload.
- Deliver repeatable installations and upgrades across cloud, on-premises and air-gapped environments.
Required qualifications
- Experience building and supporting production Kubernetes platforms, including GPU or distributed computing workloads.
- Strong software engineering skills in Python, Go or a comparable language.
- Experience automating infrastructure deployment, configuration and upgrades.
- Practical Linux, networking and storage troubleshooting skills.
- Experience with shared resource scheduling, access controls or workload isolation.
- Ability to work directly with machine-learning engineers, understand their workloads and turn recurring problems into maintained platform capabilities.
Nice to have
- Experience with Run:ai or comparable GPU scheduling and resource-management systems.
- Familiarity with distributed training and tools such as Ray or PyTorch.
- Experience delivering on-premises or air-gapped platforms, or contributing to open-source infrastructure.
Benefits
- Bonus
- Equity
- Benefits package (full-time regular employees; not for part-time employees or military fellows)
- Temporary benefits package after 60 days of employment (temporary employees)
Before you apply
- All offers are contingent on a cleared background check and possible reference check.
- Salary depends on factors such as skill set, experience, licenses and certifications, and work location.
- Speak to a talent acquisition representative for more information on the offer.
About the company
Shield AI is a venture-backed defense-tech company that builds intelligent systems to protect service members and civilians. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation technologies.