Senior Product Operations Manager, Evaluation Quality

The role

This person will own the quality bar for human evaluations by defining methodology and data analysis standards. They will work with Evaluation Operations, Applied Legal Researchers, Product, Engineering, and AI Research to produce decision-grade outputs.

What you will do

• Own the quality bar for human evaluations and produce decision-grade outputs for EPD
• Author and maintain evaluation guidelines, instructions, and databases for contract attorneys
• Standardize and streamline rubric and evaluation design into repeatable templates
• Manage contract-attorney quality including onboarding, calibration training, and inter-rater reliability
• Conduct quantitative and qualitative statistical analyses to turn raw results into prioritized signal
• Run QA on vendor and contract-attorney deliverables
• Ensure the quality bar holds across jurisdictions and non-English geographies
• Establish a standard way to analyze eval results and build operational dashboards
• Support ALR in a review step to certify evaluations before they scale
• Partner with ALR and Analytics to align human eval with online signal

Required qualifications

• 6+ years in product operations, research operations, evaluation/QA operations, or quality program management
• Track record of owning quality inputs for complex, expert-driven or human-in-the-loop work
• Experience onboarding, training, and calibrating a distributed pool of expert raters, annotators, or reviewers
• Grounding in measurement concepts including calibration, inter-rater reliability, sampling, and rubric design
• Experience running quantitative and qualitative data analyses and interpreting statistical tests
• Record of scaling and streamlining evaluation quality processes under shipping pressure
• Ability to work deeply with domain experts and translate judgment into documented standards
• Strong cross-functional coordination across Product, Engineering, Research, ALR, and data providers/vendors
• Clear communicator who can build credibility and trust with stakeholders
• Bias to action and high ownership

Nice to have

• Experience in legal tech or working with domain experts in regulated industries
• Experience owning quality across multiple markets, languages, or jurisdictions
• Built calibration, inter-rater reliability, or capability-tracking systems for annotation or evaluation pipelines
• Experience transitioning evaluation work in-house or otherwise improving evaluation ROI
• Familiarity with LLM-as-judge / automated evaluation used alongside human eval
• Early employee at a hyper-growth startup, or experience at a world-class product or platform operations org

About the company

Harvey is transforming how legal and professional services operate by combining frontier agentic AI, an enterprise-grade platform, and deep domain expertise.