Senior / Staff Software Engineer — AI Platform

The role

You will build infrastructure for running large-scale agentic workloads, synthetic data generation, and evaluation pipelines. You will design orchestration and distributed compute systems to support AI experimentation and production.

What you will do

• Design and build infrastructure for running large-scale agentic workloads
• Build scalable synthetic data generation and automated labeling systems
• Design evaluation infrastructure for measuring AI system behavior
• Build orchestration and distributed compute systems
• Develop infrastructure for agent simulation environments
• Build and operate LLM infrastructure for routing, rate limiting, and execution
• Instrument agent and model workloads for observability
• Design systems to make non-deterministic workloads reproducible and measurable
• Improve developer experience by building APIs, SDKs, and tooling
• Collaborate with research, product, and engineering teams

Required qualifications

• 5+ years building production software systems
• Experience in AI/ML infrastructure, ML platforms, distributed systems, data platforms, or backend infrastructure
• Experience operating non-deterministic AI or ML workloads in production or at significant scale
• Experience building infrastructure for experimentation, evaluation, model development, synthetic data, agentic workflows, training, inference, or production ML systems
• Strong proficiency in Python
• Experience building production-quality APIs, services, and developer tooling
• Strong background in distributed systems and cloud platforms (AWS preferred)
• Experience with workflow or distributed execution frameworks such as Prefect, Airflow, Dagster, Ray, Kubernetes, or similar systems
• Strong understanding of production system fundamentals
• Ability to reason about AI system quality beyond traditional service metrics
• Track record of leading complex engineering initiatives, influencing stakeholders, and delivering measurable impact
• Ability to work in a fast-paced environment with strong technical communication skills

Nice to have

• Experience building or operating LLM or agent infrastructure
• Experience building evaluation or experimentation platforms for LLMs, agents, or other probabilistic systems
• Experience with synthetic data generation, automated labeling, data refinement, or dataset quality systems
• Experience building reinforcement learning environments, agent simulations, benchmarks, or other environment-based evaluation systems
• Experience running large-scale distributed AI workloads across containers, Kubernetes, serverless compute, sandboxes, or heterogeneous compute environments
• Experience with LLM observability, tracing, prompt/version management, token and cost attribution, rate limiting, caching, or multi-provider routing
• Experience designing isolation and sandboxing infrastructure for executing model-generated code or tool calls safely
• Experience building shared AI platform libraries or SDKs
• Experience in hyper-growth startup environments or scaling engineering organizations
• Prior experience as a Tech Lead, Team Lead, or hands-on Engineering Manager

About the company

Snorkel AI is a frontier AI data lab that helps teams build the data and environments behind high-performing frontier and agentic AI. Founded out of the Stanford AI Lab in 2019, the company works with leading AI labs and enterprises.