Spaitial
Machine Learning Systems & Infrastructure Engineer
London. Posted 6 months ago
The role
You will build and own systems that turn raw real-world data into trained world models and reliable production endpoints. This involves designing and operating scalable training stacks, data ingestion pipelines, experiment orchestration, and model serving for large diffusion-based generative models.
What you will do
- Own and evolve ML systems for training, evaluation, and serving of large foundation models
- Improve high-throughput distributed training stacks for performance and stability
- Build end-to-end Python pipelines to turn capture sources into versioned training datasets
- Optimize underlying storage at petabyte scale
- Operate systems for launching experiments, data jobs, and production endpoints
- Maintain a launcher SDK for one-command runs
- Ship workloads with Docker and Kubernetes
- Maintain IaC and CI/CD pipelines
- Monitor, log, and alert for job performance, data-pipeline health, and cost
- Manage secrets, IAM, and network boundaries
Required qualifications
- 3+ years writing production-quality Python in a large, multi-author codebase
- Strong SWE fundamentals
- Hands-on with modern ML training stacks (PyTorch; DDP/FSDP or comparable)
- Experience debugging distributed jobs across many GPUs and nodes
- Shipped non-trivial end-to-end data pipelines at scale
- Hands-on GPU compute and performance debugging (CUDA/NCCL, GPU utilization, networking bottlenecks, profiling)
- Working knowledge of cloud environments (AWS, GCP, or Azure)
- Proficient with containers (Docker, Kubernetes)
- Comfortable reading and writing IaC (Terraform)
- Strong working knowledge of how to store and query large datasets at scale
- SQL fundamentals
- Experience with monitoring and observability tooling (Prometheus/Grafana, OpenTelemetry) and CI/CD
Nice to have
- ML systems experience
About the company
SpAItial is pioneering the next generation of World Models, pushing the boundaries of generative AI, computer vision, and simulation.