Senior AI Product Manager, Evals & Annotations

The role

You will own the Evals & Annotations product area end-to-end. This includes managing the evaluation lifecycle, setting product strategy, and building prototypes for developer workflows.

What you will do

• Own Arize's Evals & Annotations product area end-to-end
• Set the product strategy and roadmap
• Develop evaluation workflows and independently prototype solutions
• Shape evaluations for autonomous and managed agents
• Work with Engineering on technical problems regarding backend systems and data infrastructure
• Conduct multiple customer conversations each week to validate product direction
• Collaborate with cofounders and cross-functional teams on prioritization
• Drive products from problem definition through launch and iteration

Required qualifications

• Significant product management experience owning technically complex developer-facing products
• Strong understanding of LLM evaluations and AI development workflows
• Experience with developer tools, AI infrastructure, or ML platforms
• Ability to debug issues independently and navigate code
• Ability to build prototypes and technical proofs of concept
• Ability to propose technical implementations and reason through tradeoffs
• Excellent product and UX instincts
• Strong ownership, initiative, and bias toward action
• Collaborative, low-ego approach
• Experience working closely with technical customers

Nice to have

• Direct experience with LLM evaluation platforms, annotation systems, human-in-the-loop workflows, or agent orchestration
• Strong backend or platform engineering background
• Hands-on programming experience and comfort working across backend and frontend technologies
• Experience with LLM-as-a-judge, custom evaluators, experiment tracking, datasets, or feedback-driven improvement workflows
• Previous engineering experience or substantial hands-on software development experience
• Early-stage or high-growth startup experience

Benefits

• Medical
• Dental
• Vision
• 401(k) plan
• Unlimited paid time off
• Generous parental leave plan
• Mental and wellness support
• WFH monthly stipend for co-working spaces

About the company

Arize AI is the leading AI & Agent Engineering observability and evaluation platform. The company serves over 150 enterprises and Fortune 500 companies.