Research, Evals

The role

An ML evals engineer will design and build the evaluation stack at Exa. This role involves investigating how to evaluate search engines in an LLM world and building a comprehensive eval suite.

What you will do

• Write a manifesto of what perfect search means
• Design and implement evaluation frameworks that probe the limits of search
• Build scalable, reliable eval pipelines that track regressions, drift, and quality signals across billions of documents
• Create golden datasets, synthetic benchmarks, agentic tasks, and real-world test suites
• Partner with ML researchers, data engineers, infra engineers, and product to shape feedback loops

Required qualifications

• Hands-on ML experience (training, finetuning, or evaluating models)
• Strong engineering fundamentals
• Can build reliable systems (Python, Rust, distributed pipelines, GPU/cluster jobs, etc.)
• Enjoy diving into data via building eval sets, inspecting edge cases, designing creative measurement strategies

Nice to have

• ML experience related to embeddings or LLMs

Benefits

• Premium healthcare benefits (medical, dental, vision)
• Fertility benefits
• 16 weeks of fully paid parental leave for all new parents
• Monthly wellness stipend

Before you apply

• We're happy to sponsor international candidates (e.g., STEM OPT, OPT, H1B, O1, E3)

About the company

Exa is an applied AI lab building a search engine. They build massive-scale infra to crawl the web, train embedding models, and design vector databases.