Evaluations Engineer

The role

You will own leaderboards by testing new models against benchmarks in fields like law, tax, coding, and finance. You will analyze model error modes and work with the communications team to release results.

What you will do

• Evaluate new LLM model releases across the Vals AI suite of benchmarks
• Work directly with open-source and closed-source foundation model labs
• Use tools like Docent to analyze common failure modes and patterns
• Work with the social media team to post findings and results
• Add new models and maintain integrations in the model library
• Help improve and maintain benchmark infrastructure

Required qualifications

• Learning velocity
• Ownership
• Intensity
• Solution-oriented mindset

Nice to have

• Familiarity with the LLMs
• Strong engineering fundamentals
• Python expertise
• Team collaboration
• Strong work ethic
• Previous experience with benchmarking large language models, or creating benchmarks
• Previous experience working at a startup or starting your own company
• Technical writing experience and ability
• Machine learning research experience

Benefits

• Highly competitive salary and meaningful ownership
• Relocation and transportation support
• Health/dental insurance coverage
• Lunch and dinner provided
• Free snacks/coffee/drinks
• 401K plan
• Unlimited PTO
• $1,500 housing stipend (within one-mile radius)

About the company

Vals AI is building the standard for evaluating the ability of LLMs to perform real-world tasks. The founding team comes from NLP evaluation research at Stanford.