Research Scientist (Remote/US/LATAM)

The role

You will own the problem of measuring frontier model capability. You will design benchmarks, build evaluation packages, and recruit experts to validate results.

What you will do

• Turn eval targets into original benchmark designs.
• Own measurement questions including construct validity, reliability, and contamination.
• Build evaluation packages with subject-matter experts.
• Recruit, calibrate, and review a pool of experts in coding, agentic/tool-use, and STEM/reasoning.
• Act as a technical point of contact for labs.
• Translate lab requests into evaluation designs.
• Turn lab requests into sample packages and own pilots end to end.
• Help turn generalized work into public benchmarks and papers.

Required qualifications

• Research background in ML evaluation or benchmarking.
• Deep LLM/frontier-model benchmarking expertise.
• Real strength in code-model and agentic evaluation.
• Fluency with the measurement problem including construct validity, psychometrics, rubrics, pass rates, headroom, and contamination.
• Interest in the safety side of evaluation, capability elicitation, and robustness.
• Proven ability to hold a team or expert pool to a rigorous standard.
• Comfort with the full research loop.
• Fluent English

Nice to have

• Spanish a plus

About the company

Anyone AI Labs is a company focused on measuring frontier model capability.