Senior Staff Engineer, ML Ops (R4941)

The role

You will help define and build the AI Factory Reference Architecture, a Kubernetes-native platform for developing, training, evaluating and deploying AI systems. You will work with ML researchers, platform engineers and autonomy teams to make training and deployment easier for them.

What you will do

• Lead design and implementation of the AI Factory Reference Architecture
• Partner with ML researchers to support AI frameworks, foundation models, reinforcement learning and distributed training
• Design self-service AI workflows from local experimentation to large-scale distributed execution
• Build infrastructure for distributed training, simulation, inference and reinforcement learning
• Design and optimize shared GPU infrastructure across cloud and on-premises environments
• Build dataset management, experiment tracking, model versioning, evaluation and deployment capabilities
• Develop repeatable deployment solutions using Infrastructure as Code for cloud, customer-managed, sovereign and air-gapped environments
• Evaluate emerging AI infrastructure technologies and set architectural patterns
• Work with researchers, autonomy, infrastructure and product teams

Required qualifications

• Experience building Kubernetes-native AI or MLOps platforms for distributed machine learning workloads
• Deep understanding of AI training frameworks including PyTorch and Hugging Face Transformers, and distributed training techniques
• Experience operating GPU-accelerated infrastructure and distributed training systems
• Strong understanding of Kubernetes, Linux, networking, security, storage and distributed systems
• Experience with GPU scheduling concepts and large-scale AI workloads
• Experience packaging and deploying cloud-native infrastructure using Terraform and Helm
• Strong software engineering skills in Python and Golang
• Experience translating research workflows into scalable platform capabilities with ML researchers

Nice to have

• Experience with Ray or other distributed AI orchestration frameworks
• Experience with KAI, Slurm or other GPU scheduling technologies
• Experience supporting reinforcement learning, simulation-driven training, robotics or autonomy workloads
• Experience deploying and optimizing AI models for edge hardware
• Experience designing infrastructure for classified, sovereign or air-gapped environments
• Experience with OpenTelemetry, Prometheus and Grafana
• Experience contributing to or maintaining open-source infrastructure projects

Benefits

• Bonus
• Equity
• Benefits for full-time regular employees
• Temporary benefits package for temporary employees, applicable after 60 days of employment

Before you apply

• All offers are contingent on a cleared background and possible reference check
• Military fellows and part-time employees are not eligible for benefits
• Salary depends on skill set, experience, licenses and certifications, and work location

About the company

Shield AI is a venture-backed defense-tech company that builds intelligent systems to protect service members and civilians. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation technologies.