Applied AI Researcher, Benchmarking

The role

Researchers design evaluation frameworks and construct benchmarks to measure reasoning depth, interaction quality, and operational impact. They explore new paradigms for evaluating intelligent systems and establish methodologies for quantifying emergent capability.

What you will do

• Define how progress is measured
• Design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact
• Construct benchmarks that reflect real-world complexity
• Explore new paradigms for evaluating intelligent systems
• Conduct adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment
• Investigate how metrics shape model behavior
• Establish rigorous methodologies for quantifying emergent capability

Required qualifications

• Experience Designing and Running Evaluations
• Statistical and Analytical Rigor
• Experience Building with Models, Not Just Building Models
• Proven Track Record of Research Results
• Uses AI Every Day
• Strong Programming and Data Analysis Skills
• Biases Towards Showing vs Telling

Nice to have

• expertise in compound AI systems, agentic collaboration, and associated techniques (ensembling, ReAct, graph-of-thoughts, etc.)

Benefits

• 100% coverage of medical, dental, and vision insurance for employee and dependents
• Flexible time off
• Retirement and financial planning benefits, including access to pre-tax HSA, FSA, and commuter accounts, 401(k), and financial coaching resources
• Comprehensive wellness benefits, including physical fitness, mental well-being, and fertility and family-building benefits through Carrot
• Complimentary in-office lunches and snacks provided
• Access to state-of-the-art AI models, generous usage of modern AI tools, and real-world business problems
• Ownership of high-impact projects across top enterprises

About the company

Distyl is an applied AI technology company partnering with the world’s most ambitious institutions to rearchitect critical operations for the frontier of AI. They research and deploy technologies that power AI-native operations.