Software Engineering Director, Agentic Evaluations

The role

You will deliver useful and credible evaluations of agents offered by software vendors. You will lead a core team and enhance the system that performs validation of agent performance.

What you will do

• Help scope feasibility and effort for evaluating software vendor agents
• Own the full stack of the core evaluation system
• Design evaluation system primitives that generalize across software verticals
• Distill onboarding and maintenance processes into repeatable AI skills or agents
• Monitor emerging frameworks and techniques for agent evaluation
• Collaborate with data science peers to develop proprietary benchmarks
• Facilitate growth and mentorship of a core team of engineers
• Provide technical leadership and subject matter expertise in evals
• Socialize the use of evals in agent-oriented products

Required qualifications

• 10+ years of professional programming experience in backend or full-stack environments
• 2+ years of experience directly managing engineers
• Expert-level proficiency in backend development using Python, Java/Kotlin, Typescript/Javascript or Go
• Strong proficiency in associated backend frameworks like FastAPI or Node.js
• Direct experience creating evaluations against a customer-facing agent using trajectory trace data and rubrics
• Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications
• Regular use of coding agent harnesses like Claude Code, Codex, Opencode, or Pi
• BS/BA degree in related field

Nice to have

• Experience with agent tool use via direct integration or as mediated by MCP servers
• Knowledge of STATE-Bench, tau2-bench or similar benchmark frameworks
• Experience using Playwright, browser-use, Chrome DevTools MCP or others to drive user flows in the browser
• Experience with durable execution frameworks & workflow solutions, like Temporal, DBOS, Cloudflare / Vercel Workflows, or similar
• Experience with agent sandboxing using AWS E2B, Daytona, Cloudflare / Vercel Containers or similar

About the company

G2 is the world's largest and most trusted software marketplace. It has joined forces with Capterra, SoftwareAdvice, and GetApp to create a large source of online data and software insights.