Research Scientist, APEX Benchmarks

The role

You will lead the design of next generation APEX benchmarks and expert-built datasets. This role involves deciding what to measure, designing scoring rubrics, and partnering with academic and industry collaborators to scale production.

What you will do

• Decide what the next APEX benchmark should measure
• Own task taxonomy, difficulty calibration, contamination controls and statistical design
• Design expert-built datasets and grading rubrics at scale
• Set standards for reporting results including confidence intervals and inter-rater agreement
• Work with academic collaborators and industry partners to co-design benchmarks
• Publish arXiv papers, open datasets, blog posts, and conference talks
• Represent Mercor research with frontier labs, customers and the press
• Translate benchmark findings into arguments about the ROI of expert-curated data
• Partner with data operations, engineering, product and strategy to take benchmarks to production
• Track LLM evaluation literature

Required qualifications

• Strong applied or academic research background in LLM evaluation, benchmarking, NLP or a related field
• Track record of rigorous experimental design
• Ability to identify informative measurements for frontier model behavior
• Reasoning about sampling, variance, contamination and grader reliability
• Strong coding skills to build eval harnesses and run experiments
• Ability to present complex technical findings to technical and non-technical audiences
• Comfort operating in fast-moving, cross-functional environments
• Curiosity about GTM strategy, startup dynamics and the business of AI data
• Excitement to work from the office five days a week

Nice to have

• Ph.D. in machine learning, NLP or a related field
• Publications at top-tier venues (NeurIPS, ICML, ACL, ICLR)
• Experience authoring a widely-adopted public benchmark or dataset
• Industry experience on an evaluation, benchmarking or post-training team at a frontier lab
• Domain depth in finance, law, consulting, accounting, medicine or software engineering
• Experience designing rubrics, model-as-judge pipelines, or human annotation programs at scale

Benefits

• Bi-annual performance bonus structure
• Generous equity grant vested over 4 years
• Up to $15k Relocation bonus
• $10K housing bonus if you live within 0.5 miles of our office
• $1.5K monthly stipend for meals
• Free Equinox membership
• $200 monthly laundry reimbursement
• $200 monthly personal wellness reimbursement
• Health, Dental, Vision insurance
• 401(k) with company match

About the company

Mercor builds infrastructure that enables frontier AI labs and enterprises to train, evaluate, and improve AI on real-world professional work. The company is profitable and recently raised a $750 million Series D at a $20 billion valuation.