Research Engineer, Benchmarking - Member of Technical Staff

The role

You will design and build a unified system for evaluating agentic and algorithmic solutions. This involves creating a harness to measure task success, quality, and robustness using real execution and sandboxed grading.

What you will do

• Design and build a unified system for evaluating agentic and algorithmic solutions
• Mine real commits and traces and run sandboxed execution grading
• Build task suites for code search, code edit and repair, repository summarisation, and tool use
• Enforce controls against contamination, overfitting, and metric gaming
• Compare algorithmic and agentic approaches
• Lead external benchmark co-publications
• Feed results into approach decisions and customer engagements

Required qualifications

• PhD in computer science, machine learning, or a related field, or an equivalent research track record
• Authorship or co-authorship of a benchmark or evaluation paper at a recognised venue - NeurIPS Datasets and Benchmarks, ICML, ICLR, ACL
• A working understanding of how LLM and agent evaluation goes wrong
• Strong Python
• Comfort with sandboxed and distributed execution and CI
• Hands-on experience building or rigorously evaluating agentic or multi-step LLM systems

Nice to have

• Published agentic or tool-use benchmarks that use execution-based grading
• Experience running sandboxed execution grading at scale
• Open-source evaluation or harness tooling
• Familiarity with code-agent workloads such as search, edit, and repair

Benefits

• Competitive Salary
• Equity & Ownership
• Private healthcare
• Visa sponsorship
• Relocation benefits

Before you apply

• We offer Visa sponsorship and relocation benefits

About the company

Callosum is the Intelligent Systems Company building infrastructure that unifies heterogeneous compute across the full stack. They are engineers and scientists based in London.