Staff Product Manager - AI Evals

The role

You will own or support the product roadmap for agentic AI evaluations. You will partner with engineering and AI teams to define agent performance, product quality, and evaluation workflows.

What you will do

• Own or support the product roadmap for agentic AI evaluations.
• Partner with engineering and AI teams to understand evaluation needs and technical constraints.
• Define how agent performance and product quality should be assessed.
• Turn technical evaluation needs into product requirements, workflows, and acceptance criteria.
• Help operationalize evaluation workflows and success metrics.
• Use evaluation results and feedback to identify gaps and inform product decisions.

Required qualifications

• Strong product experience with AI or agentic systems, including experience building or managing agentic systems through product work.
• Experience directly building evaluations for agents or building the evaluation product, tooling, or system used to assess them.
• Ability to translate technical evaluation needs into product requirements, priorities, and workflows in partnership with engineering and AI teams.
• Ability to define or operationalize meaningful measures of agent performance and product quality.
• Strong communication, collaboration, and product judgment in an evolving, technically complex environment.

Nice to have

• Experience in cybersecurity, offensive cyber, government, defense, intelligence, or another mission-driven environment.
• Familiarity with evaluation approaches such as benchmarks, human or automated evaluation, regression testing, simulation, or analysis of agent traces.
• Experience working with products that combine autonomous systems with meaningful human judgment and oversight.

Benefits

• Medical, dental, and vision plan options.
• Life / AD&D, disability coverage options.
• Paid parental leave for eligible full-time employees.
• 12 weeks for birthing parents, 4 for non-birthing parents, 6 weeks for adoptive, foster, or intended parents through surrogacy.
• Paid holidays and flexible PTO.
• 401(k) with pre-tax and Roth options.
• HSA/FSA options, dependent care FSA.

Before you apply

• Role is limited to U.S. citizens.
• Some positions may require eligibility to obtain and maintain a U.S. Government security clearance.

About the company

Twenty Technologies industrializes offensive cyber operations for the U.S. and its allies. Headquartered in Arlington, Virginia, the company aims for American and allied primacy in cyberspace.