Assessment Design Lead

The role

The Assessment Design Lead defines what Speak measures and how it is measured across various assessment types. You will build rubrics and blueprints to ensure assessments accurately measure language proficiency and align with standards like CEFR.

What you will do

• Own Assessment Design for Curriculum Mastery, Proficiency, and Placement tests
• Define constructs, item blueprints, and rubrics
• Sign off on content validity for every assessment
• Decide operational definitions for mastery and passing scores
• Own rubric and rater guidelines for human-labeled data
• Audit items and rubrics for bias across learner subgroups
• Design and run the validity evidence plan
• Partner with Product Managers and ML Engineers on automated scoring and calibration

Required qualifications

• 4+ years designing rubrics, blueprints, and item specs for a real, shipped language assessment product
• Deep familiarity with frameworks like CEFR, ACTFL, or IELTS/TOEFL band descriptors
• Ability to identify if an item or rubric unfairly penalizes specific L1 backgrounds or accents
• Ability to translate qualitative constructs into technical specifications for ML engineers
• Comfortable running or interpreting statistics like inter-rater reliability, classical test theory, and basic IRT concepts
• Ability to act as the sign-off authority on content validity
• Uses AI tools directly in workflow
• Comfortable operating in a 0-to-1 environment

Nice to have

• Speech/pronunciation science background
• Familiarity with adaptive testing or IRT-adjacent concepts
• Experience at a large-scale language testing organization or similar high-rigor assessment environment
• Experience thriving in an EdTech startup environment
• Advanced degree (Master's or PhD) in psychometrics, measurement, applied linguistics, SLA, or a related quantitative field
• Has authored technical/validity reports or published assessment research

Benefits

• Opportunity to join a growing team with recent Series C funding
• Collaborative work environment
• Global nature with offices in San Francisco, Ljubljana, Seoul, Tokyo, and Taipei

About the company

Speak is building a human-level, AI-powered tutor that provides a conversation-first experience for language learners. The company is a leading AI company with a distributed team across several global cities.