Research Engineer, Evals

The role

You will build benchmarks, datasets, tooling, and evaluation loops to measure and improve model quality. This role involves creating domain-specific evaluations for fraud, identity, and risk workflows.

What you will do

• Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows
• Design and run offline and online evals that measure model performance on real customer tasks
• Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling
• Study where models and agents break and turn failures into better evals, datasets, and training loops
• Build reusable evaluation tools and quality building blocks
• Partner with research, engineering, product, and design to improve system quality
• Help create a culture of scientific experimentation, clear measurement, and continuous iteration
• Push the boundary of how AI systems are evaluated in regulated, adversarial, and high-consequence environments

Required qualifications

• Strong engineering fundamentals
• Willingness to review outputs, grade edge cases, curate datasets, and refine tasks

Nice to have

• Experience training, evaluating, or improving modern ML systems
• Strong programming skills and comfort working in research-heavy codebases
• Experience building benchmarks, datasets, evaluation pipelines, or quality systems
• Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas
• Ability to design clean experiments and draw reliable conclusions from noisy results
• Strong engineering judgment and a bias toward building
• Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains

Benefits

• Competitive salary
• Meaningful equity
• Platinum-level medical, dental, and vision insurance
• Unlimited PTO, sick leave, and parental leave
• Up to $100 per month in reimbursement for personal health and wellness expenses
• 401(k) plan

About the company

Variance is a small, talent-dense team in San Francisco building AI agents for risk investigations, fraud, and identity reviews.