Member of Technical Staff, Evals

The role

You will build new benchmarks and evaluation methods for AI models. You will check whether existing evaluations and datasets measure model performance correctly.

What you will do

• Design and validate new evaluations and benchmarks across model training and deployment
• Investigate model behavior and identify gaps in existing evaluations
• Measure dataset quality and identify opportunities to improve training data
• Improve evaluation correctness, reproducibility, and reliability
• Audit and improve upon public benchmarks, research papers, and open-source implementations
• Define metrics that accurately reflect model quality
• Build tools that make evaluation results accessible and useful across Magic
• Run data ablations

Required qualifications

• Experience developing model evaluations, benchmarks, or ML experiments
• Strong programming skills and experience working with machine learning systems
• Track record of owning technical projects end-to-end
• Skepticism toward results that cannot be reproduced, validated, or explained
• Experience analyzing model performance and interpreting experimental results
• Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure
• Commitment to raising the standard of rigor in AI evaluations

Benefits

• Equity as a significant part of total compensation
• 401(k) plan with 6% salary matching
• Generous health, dental, and vision insurance for you and your dependents
• Unlimited paid time off
• Visa sponsorship and relocation support for candidates moving to San Francisco
• Small, fast-moving, highly collaborative team working on frontier AI systems

Before you apply

• Visa sponsorship and relocation support are offered for candidates moving to San Francisco

About the company

Magic's mission is to build safe AGI that accelerates humanity's progress on the world's most important problems. The company combines frontier-scale pretraining, domain-specific RL, ultra-long context, and inference-time compute.