AI Voice Evaluation Specialist - 3 month contract

The role

Voice Specialists will evaluate the performance of AI models through real-time, voice-based conversations. You will compare responses from two different models using the same scenario to identify which delivers a stronger conversational experience.

What you will do

• Review an assigned scenario and roleplay a natural conversation with two different AI models.
• Conduct comparable conversations with Model A and Model B.
• Maintain approximately the same number of conversational turns with each model.
• Record your voice during each interaction.
• Evaluate each model across five defined evaluation dimensions.
• Identify and categorize relevant error clusters in the audio or conversation.
• Compare the performance of Model A and Model B based on evaluation criteria.
• Select the model that demonstrated the stronger overall performance.
• Write a detailed rationale explaining your final preference.
• Apply evaluation guidelines consistently across different scenarios.

Required qualifications

• Bachelors Degree
• Strong attention to detail and ability to notice subtle differences in conversational quality.
• Excellent listening and comprehension skills.
• Strong written communication skills.
• Ability to consistently follow detailed evaluation guidelines.
• Comfort speaking naturally and role-playing different conversational scenarios.
• Ability to compare two interactions fairly without allowing personal preferences to influence the evaluation.
• Strong critical-thinking and analytical skills.
• Reliability and consistency when completing structured evaluation tasks.

Nice to have

• Familiarity with AI assistants, voice-based AI, or conversational systems

About the company

Innodata is a global data engineering company. Their mission is to enable the responsible advancement of artificial intelligence by providing data, evaluation frameworks, and human expertise.