Research Engineer (Evals)

The role

The research engineer will own and maintain an internal benchmark suite covering single/multi-turn content and agentic safety. They will also work on research projects studying realistic agentic and LLM failure modes in the wild.

What you will do

• Own and maintain internal benchmark suite covering single/multi-turn content guardrails and agentic safety
• Build benchmarks that distinguish specific model capabilities
• Work with the product team to build evals covering core functionality of flagship models
• Build benchmarks for new features coming out of the research team
• Adapt and extend evals to new verticals and changing product data
• Work on research projects that study and quantify realistic agentic and LLM failure modes in the wild

Required qualifications

• Have built an LLM benchmark from scratch that distinguished specific model capabilities
• Have built synthetic data for post-training textual or multimodal models
• Can reproduce a published benchmark result and identify where the original methodology is fragile or misleading
• Write Python that other people can build on
• Have shipped and maintained production code
• Can write efficient LLM inference setups, including sensible orchestration of parallel calls, retries, rate-limit handling
• AI power-user fluent with frontier models and coding agents day to day

Nice to have

• Automated red-teaming experience
• Have worked across a range of agentic scaffolds and reproduced public benchmark results on them
• Strong knowledge of existing reward-model / monitoring / safety benchmarks
• One or more published papers in the evals / safety-evaluation space

Benefits

• Competitive compensation, including equity
• Flexible time off
• Relocation support if moving to Paris, available after probationary period
• Premium private health insurance
• Mental health support, including coverage for therapy
• Lunch and dinner covered when working from the office
• Learning and development support for courses, conferences, and opportunities to grow skills
• All hardware, subscriptions, tools, and services needed
• Team off-sites twice a year

Before you apply

• Intro call with Talent
• Test assignment
• Technical interview with Head of Fundamental Research
• Final interview with CEO

About the company

White Circle is an AI Safety company building the safety, reliability, and optimization layer for AI systems. They process over 100M+ API calls every month and fine-tune and train their own LLMs.