AI Product Manager, AI Platform - Quality and Evaluation

The role

You will define and execute the product strategy and roadmap for Ridgeline's AI Quality Platform. You will build evaluation frameworks, observability capabilities, and benchmarking methodologies to ensure AI capabilities are accurate and reliable.

What you will do

• Define and execute the product strategy and roadmap for Ridgeline's AI Quality Platform
• Build scalable LLM evaluation frameworks and automated evaluation harnesses
• Establish standardized offline and online evaluation methodologies, golden datasets, and quality benchmarks
• Define the roadmap for AI observability to monitor quality, latency, cost, reliability, and user outcomes
• Partner with engineering and AI researchers to build continuous regression testing and experimentation capabilities
• Drive product decisions using AI quality metrics, customer feedback, and quantitative evaluation data
• Collaborate with engineering teams to deliver platform capabilities
• Translate advances in foundation models and evaluation techniques into platform capabilities
• Prioritize product investments using customer insights, product analytics, and measurable business outcomes
• Communicate product vision, roadmap, and priorities to technical and executive stakeholders

Required qualifications

• 5+ years of product management experience building highly technical platform, infrastructure, or developer-focused products
• Demonstrated experience shipping production AI products powered by large language models
• Experience defining products that incorporate LLM evaluation frameworks, AI quality measurement, and continuous model evaluation
• Deep understanding of offline and online evaluation methodologies, model benchmarking, regression testing, and AI quality metrics
• Experience with LLM observability platforms such as Datadog LLM Observability, Langfuse, LangSmith, Arize AI, Braintrust, Galileo, or similar
• Familiarity with prompt engineering, experimentation frameworks, evaluation datasets, and LLM-as-a-judge techniques
• Experience partnering effectively with software engineers, ML engineers, AI researchers, and cross-functional stakeholders
• Strong analytical skills
• Excellent written and verbal communication skills
• Willingness to learn about cutting-edge technologies
• An aptitude for problem solving
• Ability to communicate effectively

Nice to have

• Experience building products around retrieval-augmented generation (RAG), AI agents, MCP, or model orchestration frameworks
• Experience with synthetic data generation and automated evaluation pipelines
• Experience building internal developer platforms or ML infrastructure products
• Experience in enterprise SaaS, developer platforms, or financial technology

Benefits

• Company Stock Plan
• Unlimited vacation
• Educational and wellness reimbursements
• $0 cost employee insurance plans

Before you apply

• You must be work authorized in the United States without the need for employer sponsorship

About the company

Ridgeline is headquartered in Lake Tahoe, with offices in New York, Reno, the Bay Area, and Dublin Ireland. It is recognized by Fast Company, Frost & Sullivan, and The Software Report.