Member of Technical Staff, Enterprise Evals Platform

The role

You will build verifiers, environments, and grading infrastructure for an enterprise evaluation platform. You will also define golden sets, run loss analysis, and manage optimization loops for agent systems.

What you will do

• Define golden sets by decomposing real tasks and encoding the expert quality bar
• Build verifiers over agent trajectories and outputs
• Build the eval platform that runs offline environments, task suites, and grading at scale
• Run loss analysis over production trajectories and turn failure modes into regression tests
• Run the optimization loop across models, prompts, skills, and harnesses
• Own the rollout gates that decide whether an agent change ships
• Partner with the Enterprise Platform team and the Applied AI engineers

Required qualifications

• Professional, academic, or research experience in agent engineering and evaluation
• Bachelor’s Degree in Computer Science or a related technical degree such as Electrical Engineering, Physics, Mathematics, Computer Engineering, Information Systems or Statistics
• 1–5 years in industry level software engineering
• Experience building evaluation suites for LLM or agent systems
• Familiarity with how benchmarks such as terminal-bench, tau-bench, and APEX are constructed
• Judgment about task and rubric design
• Strong software engineering fundamentals
• Ability to work independently on ambiguous, loosely specified problems

Nice to have

• experience with Harbor environments
• RL environments

Benefits

• Up to $15k relocation bonus
• $10K housing bonus (if you live within 0.5 miles of our office)
• $1.5K monthly stipend for meals
• Generous equity grant vested over 4 years
• Free Equinox membership
• $200 monthly laundry reimbursement
• $200 monthly personal wellness reimbursement
• Health, Dental, Vision insurance

About the company

Mercor builds infrastructure that enables frontier AI labs and enterprises to train, evaluate, and improve AI on real-world professional work. The company is profitable and recently raised a $750 million Series D at a $20 billion valuation.