Research Scientist, Human Data & Evaluation

The role

This researcher will turn ambiguous client or research questions into rigorous projects. You will develop methodologies, design human-data collection strategies, and analyze qualitative and quantitative data to produce insights.

What you will do

• Translate ambiguous client questions into clear research questions, hypotheses, and study designs.
• Develop research proposals and methodologies from scratch.
• Identify promising new research directions.
• Design human-data collection strategies, evaluation protocols, rubrics, annotation schemes, and experiments.
• Develop strong prompt strategies for eliciting, evaluating, and comparing model behavior.
• Determine appropriate sampling, evaluator selection, calibration, quality-control, and reliability methods.
• Design methodologies for creative and multimodal domains.
• Analyze qualitative and quantitative evaluation data.
• Synthesize findings into clear recommendations, research narratives, and new hypotheses.
• Produce client-ready reports, presentations, benchmarks, and research publications.

Required qualifications

• 3–5+ years of relevant research experience
• Master’s, PhD, or equivalent industry research depth in HCI, Human-Centered AI, ML, design research, or a related field
• Strong qualitative and quantitative research skills
• Experience in human evaluation, experimentation, data collection, UX/HCI, annotation, or model behavior research
• Proven ability to independently design and execute rigorous research
• Ability to turn data and observations into new insights, research directions, and actionable recommendations
• Comfortable analyzing data with Python and/or SQL
• Strong written communication
• Based in San Francisco, California

Nice to have

• Familiarity with modern ML and generative AI systems
• Understanding of RLHF / post-training / preference data / model evaluation
• Experience studying image, video, multimodal, creative, or agentic systems
• Experience designing prompts or systematically studying prompting/model behavior
• Experience working with large-scale human-data or annotation programs
• Experience working directly with AI research or model teams
• Strong publication record or other evidence of consistently generating original research ideas
• Startup / 0-to-1 experience

Benefits

• Medical, Dental, Vision Benefits
• 401k Matching
• Company laptop

Before you apply

• Interview with Recruiting Team
• Interview with Head of Research
• Technical interview with Data Lead
• Interview with Head of Contra Labs
• Paid Case Study and Presentation

About the company

Contra Labs is a human-centered AI lab focused on creative and multimodal outputs. It connects frontier AI labs with a global network of top creative talent through Contra.