Software Engineer, Applied AI- Evals

The role

You will build the product customers use to evaluate models, including APIs, data models, and UI. You will also own APEX launches, build failure analysis, and manage benchmarks to help customers turn eval runs into decisions.

What you will do

• Ship the APIs, data models and UI that help customers run evals, compare models and share scores
• Run every new frontier model on APEX and publish correct, defensible results to the public leaderboard
• Build failure analysis that helps customers see where their models fail
• Build dataset views, registries and versioning
• Build onboarding, roles, key management and usage views
• Join pilots with labs and enterprises and turn what they need into the roadmap

Required qualifications

• Bachelor's Degree in Computer Science or a related field
• Strong full-stack skills in a modern backend language (Python, Go or TypeScript) and a frontend framework such as React
• A track record of owning product features end to end, from idea to production
• Strong product sense that starts from what the user needs
• Experience building data-heavy products: data models, APIs and views over large volumes of results
• Excellent written and verbal communication with engineers, researchers and customers
• Comfort with ambiguity and a habit of shipping iteratively

Nice to have

• Experience with LLM evals, benchmarks or AI developer tools
• Startup experience or experience building B2B developer products

Benefits

• Bi-annual performance bonus structure
• Generous equity grant vested over 4 years
• Up to $15k Relocation bonus
• $10K housing bonus (if you live within 0.5 miles of our office)
• $1.5K monthly stipend for meals
• Free Equinox membership
• $200 monthly laundry reimbursement
• $200 monthly personal wellness reimbursement
• Health, Dental, Vision insurance

About the company

Mercor builds infrastructure that enables frontier AI labs and enterprises to train, evaluate, and improve AI on real-world professional work. The company is profitable and recently raised a $750 million Series D at a $20 billion valuation.