Product Evals Engineer, ChatGPT Learning

The role

You will lead the development of systems that evaluate response quality across ChatGPT learning experiences. You will build evaluation frameworks, define quality metrics, and develop scalable evaluation pipelines.

What you will do

• Build and evolve evaluation frameworks for quality, triggering, and information-grounded experiences within ChatGPT Learning.
• Define quality metrics and evaluation criteria for relevance, factual accuracy, source quality, coverage, freshness, citation support, safety, and educational usefulness.
• Develop scalable evaluation pipelines that combine expert human judgment, behavioral signals, model-based grading, and deterministic checks.
• Create representative datasets and benchmarks spanning student, educator, and lifelong-learning use cases.
• Design experiments that identify quality gaps, measure progress, and help teams make informed product and model decisions.
• Analyze evaluation results and user behavior to uncover systematic failure modes, emerging trends, and high-impact opportunities.
• Improve the reliability of automated evaluators through calibration, validation, and ongoing comparison with human judgments.
• Partner with engineering and research teams to translate findings into improvements across retrieval, ranking, model behavior, and product design.
• Build tools that make evaluation results accessible, interpretable, and actionable for teams across the organization.
• Help establish best practices for measuring the quality and learning impact of AI-powered search experiences.

Required qualifications

• Strong software engineer who can design reliable data pipelines, evaluation infrastructure, and internal tools.
• Understand information retrieval concepts and metrics.
• Experience working with human-labeled data, relevance judgments, annotation programs, or model-based evaluation.
• Comfortable going deep into qualitative and quantitative data to identify subtle issues and systemic patterns.
• Ability to turn ambiguous product goals into clear, testable definitions of quality.
• Communicate technical findings clearly and influence decisions across research, engineering, product, and nontechnical partners.

Nice to have

• Experience evaluating large language models, retrieval-augmented generation systems, or agentic search products.
• Familiarity with learning science, education technology, assessment design, or educational measurement.
• Experience with causal inference, online experimentation, or analysis of large-scale product telemetry.
• Knowledge of evaluation challenges involving multilingual, multimodal, or age-appropriate learning experiences.
• Experience developing or validating LLM-based graders and other automated evaluation methods.

Benefits

• Relocation assistance to new employees

Before you apply

• Background checks for applicants will be administered in accordance with applicable law

About the company

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity.