Member of Technical Staff, Evals and Post-Training

The role

You will build evaluation and measurement systems to assess AI performance on marketing tasks. These systems will serve as the foundation for post-training initiatives including benchmarks, graders, and environments.

What you will do

• Build Profound’s eval system including tasks, rubrics, LLM judges, and programmatic verifiers
• Define what a great AI Marketer run looks like and analyze trajectories
• Work with marketing experts and data labeling companies to define tasks and metrics
• Turn usage input use cases and failure modes into reliable evals
• Turn eval results and expert feedback into training signal
• Build tooling for debugging model and agent behavior
• Integrate evals into CI, launch reviews, and model selection

Required qualifications

• 2+ years building evaluation or measurement systems
• Strong Python and data skills
• Track record of shipping production systems
• Deep familiarity with LLM evaluation
• Sound experimental design
• Familiarity with modern post-training methods: SFT, preference optimization, and RL
• Strong opinions about how models and agents should behave
• Comfortable operating in a fast-paced startup environment

Benefits

• Base salary
• Equity
• Full range of benefits and perks

About the company

Profound is an AI marketing platform used by the Fortune 500 and innovators. It uses data from 2B+ real user prompts to power an AI Marketer that automates research, content, and reporting.