Product Manager, Toast IQ Evaluation

The role

You will define quality standards for Toast IQ by studying real usage and working with restaurant operators. You will build evaluation datasets, identify failure modes, and turn findings into product decisions to improve the assistant.

What you will do

• Review real Toast IQ sessions every week.
• Work directly with operators in live sessions to check evaluations.
• Group observations into failure modes and measure impact.
• Create and maintain labeled evaluation datasets.
• Validate fixes in real sessions and create regression cases.
• Lead reviewer calibration sessions for product teams.
• Surface tools that are not delivering value or intents that cannot be handled.
• Use AI coding tools to create review interfaces and automate analysis.

Required qualifications

• Worked in a restaurant or closely with operators.
• Comfortable reading hundreds of transcripts and labeling examples by hand.
• Can investigate data using a database, queries, or an AI coding tool.
• Can turn ambiguous observations into clear failure definitions or scoring rules.
• Can explain findings in language engineers, product teams, and leadership can use.
• Curiosity about AI systems and large language model failures.

Nice to have

• Familiarity with Toast's products
• Product management experience

Benefits

• Cash compensation (overtime, bonus/commissions if eligible)
• Equity
• Benefits

Before you apply

• Complete interviews and assessments on your own without real-time AI tools, recording, or transcription bots unless told otherwise.

About the company

Toast creates technology to help restaurants and local businesses succeed in a digital world. Toast IQ is a unified assistant that orchestrates specialized agents across the Toast ecosystem.