Research Engineer - Benchmarks

San Francisco HQ; Toronto Hub, Remote. Posted a month ago

The role

You will design and run automated, public benchmarks of data providers to measure accuracy, coverage, freshness, speed, and cost. You will own the entire process from datasets and ground truth to the weekly release and integration into Alexandria.

What you will do

Design and run head-to-head benchmarks of data providers against verified ground truth
Build and maintain test datasets and ground truth
Own the automated harness and the weekly benchmark release
Measure accuracy, coverage, freshness, speed, and cost
Work with marketing to ship public leaderboard pages
Close the loop into Alexandria so benchmark results change provider selection
Expand the categories benchmarked inside Alexandria and beyond

Required qualifications

4+ years in ML, research engineering, or data engineering
Experience with evaluation or benchmark work you've shipped
Strong in Python and API integrations
Comfortable with messy vendor APIs, rate limits, and inconsistent schemas
Knowledge of building test sets and scoring methods including sampling, labeling, and inter-rater agreement
Ability to write findings clearly for engineers, marketers, and vendors
Authorized to work in the US or Canada

Nice to have

Experience at a frontier lab, a data company like Scale, Surge, micro1 or Mercor, a third-party benchmark org, or a public benchmark project
OSS contributors

Benefits

Competitive equity
15 days mandatory PTO, more after 24 days
12 weeks fully paid parental leave
$100 USD/month wellness stipend
$1,000 USD/year Learning & Development expense
Team offsites
3 paid months sabbatical after 4 years
Medical, dental, and vision (US)
Life & Disability insurance (US and Canada)
Virtual care and health guide (US)
Mental health support (US and Canada)
Fertility and family building (US and Canada)

Before you apply

Must be already authorized to work in the US or Canada
US candidates: H-1B transfers, F-1 OPT and STEM OPT, TN, and E-3 supported
Canada candidates: work permit sponsorship considered case by case
Process includes a 40-hour paid work trial

About the company

Firecrawl is a company that turns the web into data AI agents can use via an API. They are building Alexandria, a large repository of knowledge.