Senior / Staff ML Ops Engineer

The role

You will build and improve the training infrastructure for machine learning on Kubernetes, including GPU scheduling and distributed jobs. You will also build developer tools, data layers and observability so researchers can run experiments reliably.

What you will do

• Build and evolve training infrastructure on Kubernetes, including GPU scheduling, autoscaling and multi-node distributed jobs
• Build developer-facing CLIs, SDKs, job submission tools and templates
• Shorten time to first training run and speed up iteration before jobs reach the cluster
• Evaluate and promote ML tooling and frameworks
• Strengthen dataset versioning and high-throughput loading of large multimodal sensor data
• Turn one-off Python into tested, documented, observable tools and services
• Make experiments traceable with dashboards, a model registry and lineage
• Build CI/CD for models alongside autonomy and simulation
• Build observability across the ML stack, including utilization, failures and cost
• Improve docs, onboarding and support, and drive adoption of the platform

Required qualifications

• 5+ years of software or infrastructure engineering, including tools used by other engineers and ML or data-intensive production systems
• Excellent Python and experience designing APIs and CLIs
• Practical AWS depth: object storage at scale, IAM, GPU compute, networking, cost management
• Infrastructure as code such as Terraform, Pulumi or similar
• Distributed training in PyTorch (DDP, FSDP or similar)
• Experience with experiment tracking and model registry tooling
• Fluency with containers, CI/CD and modern build systems, including large monorepos
• Ability to influence without authority and persuade senior engineers
• A collaborative approach to co-owning systems
• Strong product instincts, strong writing, and comfort working autonomously in ambiguous situations
• Passion for self-driving technologies and frontier AI

Nice to have

• Internal developer platform, research platform or developer experience work
• Large-scale distributed GPU training with NCCL and cluster networking
• High-throughput loading of LiDAR or camera data, including Parquet or WebDataset
• Workflow and scheduling systems such as Argo Workflows, Ray, Flyte, Kubeflow or Slurm
• Build-system depth such as Bazel, including remote caching
• Simulation infrastructure or large-scale batch evaluation pipelines
• Background in ML, robotics or autonomous systems infrastructure
• Experience in security- and IP-sensitive production environments
• Open-source contributions to ML infrastructure or developer tools

Benefits

• Competitive compensation and equity awards
• Medical, dental and vision coverage (full-time employees only)
• Unlimited vacation
• Flexible hours and work-from-home support
• Daily drinks, snacks and catered meals when in office
• Regular team building activities and social events, on-site, off-site and virtual

Before you apply

• The posting does not state visa sponsorship or work authorization rules

About the company

Waabi is a technology start-up led by Raquel Urtasun that works on Physical AI, including commercial autonomous trucks and robotaxis. It has offices in Toronto, San Francisco, Dallas and Pittsburgh.