AI QA Trainer - LLM Evaluation - Freelance Project

The role

AI QA trainers will evaluate large-scale language models to improve reasoning and reliability. You will converse with models on real-world scenarios, verify factual accuracy, and design test plans.

What you will do

• Converse with the model on real-world scenarios and evaluation prompts
• Verify factual accuracy and logical soundness
• Design and run test plans and regression suites
• Build clear rubrics and pass/fail criteria
• Capture reproducible error traces with root-cause hypotheses
• Suggest improvements to prompt engineering, guardrails, and evaluation metrics
• Partner on adversarial red-teaming, automation, and dashboarding

Required qualifications

• Clear, metacognitive communication

Nice to have

• Bachelor’s, master’s, or PhD in computer science, data science, computational linguistics, statistics, or a related field
• Shipped QA for ML/AI systems
• Safety/red-team experience
• Test automation frameworks (e.g., PyTest)
• Hands-on work with LLM eval tooling (e.g., OpenAI Evals, RAG evaluators, W&B)
• Evaluation rubric design
• Adversarial testing/red-teaming
• Regression testing at scale
• Bias/fairness auditing
• Grounding verification

Before you apply

• As a contractor you’ll supply a secure computer and high-speed internet

About the company

agency