Member of Technical Staff, Evals

The role

You will design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, and multimodal models. You will partner with researchers and domain experts to develop reward models, verifier systems, and data-quality techniques.

What you will do

• Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and reinforcement-learning environments.
• Develop reward models, programmatic verifiers, graders, and other feedback systems.
• Research what makes evaluations representative, difficult, reliable, and resistant to shortcutting or reward hacking.
• Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques.
• Run fast, rigorous iteration loops to prototype, evaluate, and diagnose failure modes.
• Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing.

Required qualifications

• PhD in ML/AI, computer science, data science, or related fields (or equivalent research experience in industry).
• Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM.
• Strong Python skills.
• Experience building scalable software.
• Experience working with agents.
• Strong knowledge of frontier AI: benchmarks, eval techniques, agent harnesses, post-training recipes, data shapes.
• Comfort operating in an ambiguous, fast-moving environment with substantial ownership.

Nice to have

• Builders who enjoy tinkering with agents and shipping high-quality software, benchmarks, or datasets (e.g. a strong GitHub profile / OSS contributions, or product portfolio).

Benefits

• Equity in a fast-growing company
• 401(k) match
• Competitive compensation
• Financial coaching
• Paid parental leave
• Fertility benefits
• Parental coaching
• Medical, dental, and vision
• Mental health support
• $500 wellness stipend
• $2,000 learning stipend
• Commuting support, free lunch, and gym in our SF office

About the company

Handshake's mission is to organize expert human knowledge to advance the AI economy. Handshake AI works directly with frontier labs on data, evaluation, and post-training challenges.