Principal Research Scientist

The role

You will own the research agenda for how AI agents are evaluated and turn results into product features. You will design realistic evaluations, develop measurement approaches, and mentor engineers and researchers.

What you will do

• Own the evaluation methodology and set standards for AI agent evaluation.
• Work with engineers and product managers to build and improve evaluations for current products.
• Design realistic environments and task sets for AI agents.
• Develop scoring approaches including automated checks, LLM-based judges and human review.
• Validate evaluations by auditing tasks and graders.
• Present and defend evaluation findings to buyers, customers, partners, and the research community.
• Manage and guide researchers and engineers on evaluation design and measurement.
• Set and pursue a research agenda.
• Follow new benchmarks and papers to translate them into product decisions.
• Write methodology documentation, papers, and public posts.

Required qualifications

• Experience working with evaluation design or dataset curation in multi-agentic systems
• 5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents
• A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics or a related field, or equivalent applied research experience
• A track record of original ideas in machine learning shown through first-author publications or projects led end-to-end
• The ability to own a research agenda
• A track record of shipping ML work in a product setting
• Experience setting technical direction for others
• Familiarity with how AI systems are benchmarked
• Sound statistical judgment
• Good judgment about trade-offs
• Clear writing and ability to explain methodology to non-specialists

Nice to have

• 2+ years experience managing small to medium teams
• Research experience in agent evaluation, user simulation, or human and AI-based judging
• Experience with conversational, voice, retrieval-based or tool-using AI systems
• Contributions to open-source evaluation tools or public leaderboards
• Familiarity with business software workflows
• A considered view on the effects of AI technology on the people and markets it reaches
• Publications at ACL, EMNLP, NeurIPS, ICML, ICLR, COLM or similar venues

About the company

G2 is the world's largest and most trusted software marketplace. It has joined forces with Capterra, SoftwareAdvice, and GetApp to create a large source of online data and software insights.