Member of Technical Staff - Evaluations

The role

You will work on the internal eval suite and infrastructure end to end. You will design held-out evals for capabilities like agentic workflows, computer use, and cybersecurity.

What you will do

• Work on the internal eval suite and infrastructure to keep it reliable, optimized, and reproducible at scale.
• Design held-out evals for priority capabilities such as agentic workflows, computer use, and cybersecurity.
• Build evals targeted at small edge models.
• Read generated outputs to debug unexpected scores.
• Turn customer and open-source feedback into environments usable for data, training, and evaluation.
• Partner with other teams to design evals for their specific needs.
• Release open-source evals, write papers, and publish failure-mode reviews of popular evaluations.

Required qualifications

• Unusually strong eval design skills to turn high-level descriptions into accurate, hard-to-game evaluations.
• Experience building recognizable evals that hold up on coverage, reproducibility, contamination handling, and signal.
• At least one eval built from scratch, such as a benchmark, an eval paper, or real contributions to lm-eval-harness, lighteval, OLMES, HELM, or Inspect.
• Fluency with contamination, saturation, judge bias, and variance.
• Hands-on experience running and managing evals in a team.
• The habit of debugging unexpected scores by reading individual generated answers.
• Software engineering strong enough to own eval infrastructure and debug inference and infra failures at scale.

Nice to have

• Model training experience
• Publications or research background
• Deep systems expertise
• Familiarity with what small edge models do and don't do well

Benefits

• Competitive base salary plus equity
• Unlimited PTO
• Company-wide Refill Days
• Comprehensive health coverage
• Paid leave for pregnancy, parenting, medical needs, and caring for family

About the company

Spun out of MIT CSAIL, Liquid AI builds general-purpose AI systems that run efficiently across deployment targets. They partner with enterprises across consumer electronics, automotive, life sciences, and financial services.