Senior Data Labeler

The role

You will build and lead a Data Labeling team from scratch. You will design evaluation frameworks and run labeling operations for safety evals, RLHF, and benchmarks.

What you will do

• Build and lead the Data Labeling team through hiring, coaching, and performance management
• Define annotation guidelines, quality standards, and evaluation frameworks
• Develop quality assurance processes, calibration sessions, and auditing systems
• Partner with AI researchers and engineers to translate research objectives into labeling workflows
• Prioritise labeling projects based on business and research needs
• Monitor operational metrics including quality, consistency, throughput, and cost
• Improve annotation tooling, automation, and workflow efficiency
• Lead complex AI evaluation projects including safety, preference ranking, RLHF, policy evaluation, and benchmark creation
• Analyse disagreement patterns and edge cases to improve guidelines and model performance
• Manage vendor relationships and ensure consistent quality across distributed teams

Required qualifications

• Experience leading data annotation or AI evaluation teams
• Strong operational and people management skills
• Understanding of AI model evaluation, LLM behaviour, and modern annotation workflows
• Ability to design scalable processes without sacrificing quality
• Clear communication across technical and non-technical teams
• Ability to thrive in fast-moving startup environments

Nice to have

• Managed annotation programs for LLMs, generative AI, or machine learning
• Experience with RLHF, preference data collection, safety evaluations, or benchmark creation
• Worked in Trust & Safety, AI Safety, Content Moderation, or ML Ops
• Managed distributed or global annotation teams
• Experience with vendor management and outsourcing operations
• Familiarity with prompt engineering and AI safety policies
• SQL, Python, or data analysis experience
• Experience building internal annotation platforms or workflow automation
• Background in linguistics, cognitive science, machine learning, or data operations

Benefits

• Competitive compensation, including equity
• Flexible time off
• Relocation support if moving to Paris, available after probationary period
• Premium private health insurance
• Mental health support, including coverage for therapy
• Language lessons to help improve English or French
• Lunch and dinner covered when working from the office
• Learning and development support for courses, conferences, and opportunities
• All hardware, subscriptions, tools, and services needed
• Team off-sites twice a year

Before you apply

• Role involves overseeing projects that may include offensive, harmful, violent, sexual, or otherwise disturbing content
• Intro call
• Take-home test assignment
• Final conversation with CEO
• Submit application in English

About the company

White Circle is an AI Safety company building the safety, reliability, and optimization layer for AI systems. They process over 100M+ API calls every month and fine-tune their own LLMs.