Spaitial

Machine Learning Systems & Infrastructure Engineer

London. Posted 6 months ago

The role

You will build and own systems that turn raw real-world data into trained world models and reliable production endpoints. This involves designing and operating scalable training stacks, data ingestion pipelines, experiment orchestration, and model serving for large diffusion-based generative models.

What you will do

  • Own and evolve ML systems for training, evaluation, and serving of large foundation models
  • Improve high-throughput distributed training stacks for performance and stability
  • Build end-to-end Python pipelines to turn capture sources into versioned training datasets
  • Optimize underlying storage at petabyte scale
  • Operate systems for launching experiments, data jobs, and production endpoints
  • Maintain a launcher SDK for one-command runs
  • Ship workloads with Docker and Kubernetes
  • Maintain IaC and CI/CD pipelines
  • Monitor, log, and alert for job performance, data-pipeline health, and cost
  • Manage secrets, IAM, and network boundaries

Required qualifications

  • 3+ years writing production-quality Python in a large, multi-author codebase
  • Strong SWE fundamentals
  • Hands-on with modern ML training stacks (PyTorch; DDP/FSDP or comparable)
  • Experience debugging distributed jobs across many GPUs and nodes
  • Shipped non-trivial end-to-end data pipelines at scale
  • Hands-on GPU compute and performance debugging (CUDA/NCCL, GPU utilization, networking bottlenecks, profiling)
  • Working knowledge of cloud environments (AWS, GCP, or Azure)
  • Proficient with containers (Docker, Kubernetes)
  • Comfortable reading and writing IaC (Terraform)
  • Strong working knowledge of how to store and query large datasets at scale
  • SQL fundamentals
  • Experience with monitoring and observability tooling (Prometheus/Grafana, OpenTelemetry) and CI/CD

Nice to have

  • ML systems experience

About the company

SpAItial is pioneering the next generation of World Models, pushing the boundaries of generative AI, computer vision, and simulation.

Machine Learning Systems & Infrastructure Engineer at Spaitial | JobZoey