Special Project Lead

The role

The Special Project Lead will own end-to-end project delivery for client engagements including post-training dataset creation and benchmarking. You will act as an operator between clients, creative experts, and internal engineering teams to ship rigorous results.

What you will do

• Develop annotation protocols, rubrics, and evaluation designs.
• Manage scope, evaluator staffing, quality routing, timelines, and client relationships.
• Ensure data quality and inter-rater reliability.
• Perform qualitative and quantitative analysis for findings, recommendations, and reports.
• Partner with engineering to build and improve tooling.
• Create case studies and benchmark publications.

Required qualifications

• 3+ years designing studies, running evaluations, analyzing data, and presenting findings.
• Experience with data labeling, annotation, human-in-the-loop systems, or AI and ML evaluation workflows.
• Track record managing several concurrent projects, including annotation, evaluation, or QA work.
• Based in NYC.
• In office 5 days a week (Williamsburg).

Nice to have

• Strong research background, qualitative and quantitative.
• Master's or PhD in HCI, cognitive science, psychology, design research, or a related field.
• Statistical methods for evaluation work: inter-rater reliability, sampling, regression, significance testing.
• RLHF, LLM evaluation, or benchmark design.
• Ethnography, UX research, or design-study methods.
• Startup or 0-to-1 experience.

Benefits

• Medical, dental, vision
• 401k matching
• Company laptop on your start date

Before you apply

• Recruiting team interview
• CEO and co-founder interview
• Technical interview, research and methods focus
• VP of Product interview
• Paid case study and presentation

About the company

Contra Labs is a human-centered research lab studying how creatives use AI and how tools can serve creative work. They connect a global network of creative talent with frontier AI labs and product companies.