AI Agent Evaluator

The role

You will evaluate how well AI assistants understand users, use personal context, and deliver helpful conversations. You will have natural, back-and-forth conversations with two versions of an AI assistant and compare your experiences.

What you will do

• Write prompts about questions, interests, or needs relevant to you.
• Have natural, multi-turn conversations with two AI assistants.
• Include questions that draw on information or activity from your connected services, where relevant.
• Compare how effectively each assistant uses available information and connects relevant context.
• Assess whether responses feel helpful, natural, trustworthy, and safe.
• Evaluate how each assistant responds to your feedback and corrections.
• Provide clear explanations supporting your evaluations.

Required qualifications

• Fluency in one of the supported languages/locales: Spanish — Latin America, Arabic, Portuguese — Brazil, Japanese — Japan, Turkish — Turkey, Indonesian — Indonesia, Hindi — India, Korean — Korea, Fren
• Strong reading comprehension and the ability to explain your feedback clearly.
• Attention to detail and sound judgment when comparing conversation experiences.
• Willingness to interact with AI assistants using relevant personal context and connected services.
• Recent activity across a qualifying combination of digital services.
• Completion of the project’s eligibility verification and corresponding expert requirements.

Nice to have

• Experience using AI assistants for everyday questions, planning, or research.
• Previous experience in AI evaluation, user research, or quality assurance.
• Ability to identify subtle differences in relevance, personalization, and conversation quality.

Before you apply

• Complete a brief verification to confirm whether the digital services you currently use and your recent activity meet the project’s requirements.
• Meeting the required activity threshold is necessary for participation.
• Completing verification does not guarantee placement.

About the company

Anyone AI creates high-quality training and evaluation data for frontier AI models.