Member of Technical Staff, ML Platform (Evals)

The role

This role builds the evaluation platform that measures how well Runway's AI models perform. The person will design systems that generate and score model samples, track results across checkpoints, and set standards for trustworthy model comparisons. The work is embedded with research teams and sets technical direction for evals company-wide.

What you will do

• Own the evaluation platform end to end, including tooling to generate, annotate, review and adapt models at frontier scale
• Define CLIs, APIs, GUIs and storage layers for a fast, collaborative evaluation process
• Work with research teams on video, image, audio, agents and robotics to decide what to measure and build it
• Set standards for reproducibility, metric definitions, reporting formats and when a result is trustworthy enough to act on
• Support adoption of the platform across training, production model serving and other research efforts
• Contribute to tools and systems that help train and serve frontier models

Required qualifications

• 5+ years building ML infrastructure or data platforms in production, with some time on evaluation, experimentation or benchmarking systems
• Strong Python and PyTorch
• Hands-on experience running large batch GPU workloads on Kubernetes
• Experience designing data pipelines and storage for large volumes of media or model outputs, with versioning and reproducibility
• Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
• Comfort building internal tools end to end, from the command line to the browser
• Ability to lead a broad technical area: gather requirements, set direction, make tradeoffs and drive a roadmap
• Familiarity with the full model development lifecycle: data, training, evaluation, serving
• Self-starter who can work embedded with research teams and move fast
• Strong systems thinking and a pragmatic approach to production reliability
• Humility and open-mindedness

Nice to have

• Track record of building evaluation suites for generative models
• Hands-on work with LLM- or VLM-as-judge pipelines
• Experience with online experimentation platforms and connecting offline metrics to product outcomes
• Prior work evaluating agents or robotics policies

Benefits

• Salary is one part of the overall compensation package
• Pay equity and internal equity with peers are considered in salary decisions
• Access to best-in-class models, agents, GPUs, storage and cloud services

Before you apply

• Salary range applies to candidates in the U.S.; ranges outside the U.S. may differ and will be communicated

About the company

Runway is building AI world models to simulate the world by merging art and science, aiming to change how stories are told and how scientific progress is made. The company describes its team as creative, open-minded, caring and ambitious.