Member of Technical Staff, Human Data Quality Engineer

The role

You will own upstream data quality for LLM post-training and evaluation. You will design automated QA methods and build reusable pipelines to deliver high-quality data to post-training teams.

What you will do

• Own upstream data quality for LLM post-training and evaluation
• Analyze expert-developed datasets and operationalize quality standards
• Partner with research and post-training teams to translate requirements into measurable quality signals
• Provide actionable feedback to external data vendors
• Design, validate, and scale automated QA methods
• Build reusable QA pipelines for model training and evaluation
• Monitor and report on data quality over time

Required qualifications

• Strong engineering fundamentals with experience building data pipelines, QA systems, or evaluation workflows
• Detail-oriented with an analytical mindset
• Solid understanding of how data quality impacts training (SFT and RL) and evaluation
• Experience designing and validating automated quality checks
• Ability to work autonomously and collaborate with researchers, engineers, and operations partners
• Proficiency in Python and building ML / LLM workflows
• Ability to debug and write scalable code
• Experience working with large datasets and automated evaluation or quality-checking systems
• Familiarity with how LLMs work and how models are trained and evaluated
• Excellent communication skills

Benefits

• Salary and equity
• Stock options
• Comprehensive medical, dental, vision, and life
• Annual wellness allowance
• Lunch and dinner provided in the office daily
• 22 weeks paid parental leave
• Unlimited paid time off in the U.S.
• 30 days paid time off in the U.K.
• Visa sponsorship support
• Regular off-sites, happy hours, and team celebrations

Before you apply

• This position may require access to technology or source code subject to the U.S. Export Administration Regulations
• Any offer of employment may be conditioned on the Company's ability to provide access to technology or source code in compliance with U.S. export control laws

About the company

Reflection is a research lab making intelligence open and accessible for everyone to use, customize, and build on. They build open models to help shape the future of AI.