Research Engineer – Benchmarking

The role

You will own benchmarking pipelines, evaluation systems, and failure analysis workflows. This work informs how frontier language models are trained and improved.

What you will do

• Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning
• Build and operate LLM evaluation systems end-to-end
• Run systematic failure analysis on model outputs
• Create and refine rubrics, automated evaluators, and scoring frameworks
• Quantify data usability, quality, and impact on key benchmarks
• Work with AI researchers, applied AI teams, and data producers
• Operate in a high-iteration research setting with strong ownership

Required qualifications

• Strong applied research background, with focus on model evaluation, benchmarking, and/or failure analysis
• Strong coding skills and hands-on experience with ML models and evaluation code
• Solid grasp of data structures, algorithms, and backend systems
• Comfort with APIs, SQL/NoSQL, and cloud platforms for running and storing eval results
• Ability to reason about model behavior, experimental results, and data quality from evals and failure analyses
• Excitement to work in person in San Francisco five days a week

Nice to have

• Industry experience on a post-training or evaluation/benchmarking team
• Publications at top-tier venues (NeurIPS, ICML, ACL), especially in evaluation or benchmarking
• Experience building or running LLM evaluations, benchmarks, or failure-analysis pipelines
• Experience with synthetic data generation, rubric design, or RL-style workflows that use evals for reward shaping
• Work samples or code (e.g., eval frameworks, benchmark suites, failure-analysis reports or tooling)

Benefits

• Bi-annual performance bonus structure
• Generous equity grant vested over 4 years
• Up to $15k Relocation bonus
• $10K housing bonus (if you live within 0.5 miles of our office)
• $1.5K monthly stipend for meals
• Free Equinox membership
• $200 monthly laundry reimbursement
• $200 monthly personal wellness reimbursement
• Health, Dental, Vision insurance

About the company

Mercor builds infrastructure that enables frontier AI labs and enterprises to train, evaluate, and improve AI on real-world professional work. The company is profitable and recently raised a $750 million Series D at a $20 billion valuation.