Director, Evaluations

The role

The Director of Evaluations will build, lead, and grow the Evaluations Team. You will define the evaluation strategy and roadmap to ensure capability and safety claims for Scientist AI and frontier LLMs are trusted.

What you will do

• Define LawZero's evaluations strategy and roadmap.
• Build the Evaluations Team to roughly 8-10 people within 3-6 months.
• Operate the team independently of research and product streams.
• Design novel benchmarks for Scientist AI and frontier LLMs.
• Oversee design and construction of new datasets, tasks, and environments.
• Lead evaluation of the Scientist AI when deployed as a guardrail.
• Establish and lead automated and manual red-teaming programmes.
• Lead construction of internal tooling and infrastructure for evaluations.
• Support research and product streams with internal evaluation requirements.
• Own public communication of evaluation results including technical reports and publications.

Required qualifications

• Advanced degree (MSc or higher) in machine learning, computer science, or a closely related field.
• 10+ years of experience in machine learning.
• At least 5 years in a leadership role building or scaling technical teams working on real-world ML products.
• Hands-on expertise in designing and running large-scale evaluations of LLMs or other frontier ML systems across capabilities, safety, and adversarial robustness.
• Track record of building evaluation datasets, benchmarks, or interactive environments from scratch.
• Strong written and verbal communication skills.
• Comfortable operating in a research-driven, fast-moving environment with significant ambiguity.

Nice to have

• Experience leading red-teaming exercises (automated, manual, or both) and working with third-party evaluation or red-teaming partners
• Experience working with third-party partners for benchmark and dataset creation
• Experience releasing open-source datasets, benchmarks, or evaluation tooling
• Familiarity with current AI safety policy and standards work (UK AISI, CAISI, NIST, EU AI Act, etc.)
• Experience contributing to or coordinating with external safety institutes, grant funders, or government bodies

Benefits

• Comprehensive health benefits
• A minimum of 20 days vacation per year upon start
• A minimum retirement savings employer contribution of 4%
• Generous flexible benefits designed to contribute to your well-being
• A team of passionate experts in their field
• A collaborative and inclusive work environment

About the company

LawZero is a non-profit building safe-by-design AI systems. Based in Montreal, its research aims to build non-agentic AI that learns primarily to understand the world.