Lead Product Data Scientist, Agents

The role

You will own and advance the measurement and data science strategy for AI agents including Composer and Customer Agent. You will develop measurement frameworks, evaluate agent performance, and partner with cross-functional teams to influence product and business strategy.

What you will do

• Own agent measurement end to end across Composer and Customer Agent
• Define successful agent performance by partnering with Product and Engineering
• Build scalable agent evaluation frameworks using conversation data and LLM-based classification
• Measure causal impact and incremental value through experimentation and advanced statistical methods
• Connect agent performance to business strategy and evaluate pricing and monetization
• Establish trusted metrics and analytical standards
• Lead with an AI-first approach to analytics using coding agents and LLMs
• Build scalable, self-service analytics capabilities for stakeholders
• Partner with Engineering and Analytics Engineering to develop data pipelines and dbt models
• Lead the technical direction of agent analytics and mentor other data scientists

Required qualifications

• 10+ years of experience in data science, product analytics, applied statistics, or a related quantitative discipline
• Recent hands-on experience measuring AI, machine learning, or LLM-powered products in production
• Deep expertise in product measurement, experimentation, statistical modeling, and causal inference
• Experience evaluating AI systems in production
• Highly proficient in SQL and Python or R
• Experience developing production-quality analytical solutions
• Familiarity with dbt and modern analytics engineering practices
• Demonstrated technical leadership in setting direction and mentoring data scientists
• Degree in Statistics, Mathematics, Computer Science, Economics, Finance, Engineering, or another quantitative discipline, or an MBA with a quantitative emphasis

Nice to have

• Experience measuring and optimizing AI agents, conversational AI, or LLM-powered products in a high-growth B2B SaaS or product-focused technology company
• Hands-on experience with agent observability and evaluation platforms such as Braintrust, LangSmith, or Arize
• Experience with usage-based or consumption-based pricing, credit models, AI infrastructure costs, or LLM unit economics
• Experience building governed metrics, semantic layers, metric trees, or standardized measurement frameworks
• Experience applying classification, clustering, or other advanced analytical methods to unstructured text or agent interactions
• Experience developing internal AI agents or AI-powered analytics tools
• Familiarity with marketing attribution, incremental measurement, or evaluating the impact of AI-driven customer engagement
• Background in marketing technology, ecommerce, CRM, customer-service software, or other data-driven SaaS products

Benefits

• Annual cash bonus plan
• Equity
• Sign-on payments
• Health, welfare, and wellbeing benefits

About the company

Klaviyo empowers creators to own their destiny by making first-party data accessible and actionable. They focus on nurturing personalized experiences in ecommerce and beyond.