Senior Research Scientist, Model Evaluation

The role

You will create next-generation evaluation methods and infrastructure to measure LLM progress. This involves building new evaluation benchmarks and scalable tools to analyze model performance.

What you will do

• Create ambitious new evaluation benchmarks that push the limits of what models can accomplish.
• Work on highly cross-functional teams to translate model feedback into trustworthy, repeatable evaluations.
• Conduct research to advance the state-of-the-art in LLM evaluation methods.
• Train LLM judges.
• Refine LLM-based data synthesis pipelines.
• Improve evaluation efficiency.
• Build scalable and reusable tools for digging into model performance.

Required qualifications

• Enjoy rapidly building prototypes that demonstrate the boundaries of what LLMs are capable of.
• Developed resources to measure LLM capabilities.
• Spent dozens of hours reviewing complex data and LLM outputs to ensure high data quality.
• Obsessive about rigorously measuring AI capabilities and ensuring measurements align with capabilities.
• Strong software engineering skills.

Benefits

• Weekly lunch stipend of $75/£75 or equivalent in local currency.
• Full health and dental benefits, including a separate budget for mental health.
• RRSP matching, 401K, Pension Scheme.
• 100% Parental Leave top-up for up to 6 months, for either parent.
• Annual enrichment benefits for arts & culture, fitness/wellness, quality time, and a workspace improvement credit.
• Education & learning stipend for conferences, courses, and coaching.
• 6 weeks of paid vacation (30 working days!).
• Budget for traveling to other offices if remote, plus an annual company offsite.
• Daily lunch program, snacks, and regular community and social events for those in the office.
• Co-working benefit for those not near an office.
• $500 home office stipend.

Before you apply

• May use AI-enabled tools to screen and assess applicants.

About the company

Cohere is a security-first enterprise AI company that builds foundation AI models and end-to-end products. It is a global technology company headquartered in Toronto.