AI Voice Evaluation Specialist

The role

Voice Specialists evaluate the performance of AI models through real-time, voice-based conversations. You will roleplay scenarios with two different models to compare their responses based on defined quality criteria.

What you will do

• Review an assigned scenario and roleplay a natural conversation with two different AI models.
• Conduct comparable conversations with Model A and Model B.
• Maintain approximately the same number of conversational turns with each model.
• Record your voice during each interaction.
• Evaluate each model across five defined evaluation dimensions.
• Identify and categorize relevant error clusters in the audio or conversation.
• Compare the performance of Model A and Model B.
• Select the model that demonstrated the stronger overall performance.
• Write a detailed rationale explaining your final preference.
• Apply evaluation guidelines consistently.

Required qualifications

• Bachelors Degree
• Strong attention to detail and ability to notice subtle differences in conversational quality.
• Excellent listening and comprehension skills.
• Strong written communication skills.
• Ability to follow detailed evaluation guidelines consistently.
• Comfort speaking naturally and roleplaying different conversational scenarios.
• Ability to compare two interactions fairly without allowing personal preferences to influence the evaluation.
• Strong critical-thinking and analytical skills.
• Reliability and consistency when completing structured evaluation tasks.

Nice to have

• Familiarity with AI assistants, voice-based AI, or conversational systems

About the company

Innodata is a global data engineering company. Their mission is to enable the responsible advancement of artificial intelligence by providing data, evaluation frameworks, and human expertise.