Product Manager, Evals

The role

You will partner with research, product, GTM, and engineering to oversee the quality and completeness of evaluation data. You will manage relationships with external vendors and make strategic decisions on acquiring important data for search model evaluations.

What you will do

• Build specific, localized eval feedback and signal
• Write a manifesto of what perfect search means
• Define metrics and success criteria for our evaluation data suites
• Partner with ML researchers, data engineers, infra engineers, and product to shape data loops

Required qualifications

• Hustler in negotiating, securing, and managing relationships with external vendors
• Care and excellence in writing
• High ownership and attention to detail
• Strong numerical literacy and technical background
• CS/EE degree, production engineering experience, or equivalent depth earned on the job

Benefits

• Premium healthcare benefits (medical, dental, vision)
• Fertility benefits
• 16 weeks of fully paid parental leave for all new parents
• Monthly wellness stipend

Before you apply

• We're happy to sponsor international candidates (e.g., STEM OPT, OPT, H1B, O1, E3)

About the company

Exa is an applied AI lab building a search engine. They build massive-scale infra to crawl the web, train embedding models, and design vector databases.