Innodatainc
AI Agentic Workflow Reviewer
In Office - San Jose, California. Posted a month ago
The role
Reviewers evaluate complex agentic AI workflows to assess task success, safety, and policy adherence. You will work in isolated test environments using structured rubrics to judge agent behavior.
What you will do
- Review multi-step agent task trajectories against rubric-based criteria
- Evaluate whether an agent respected user agency and informed consent
- Apply privacy guardrails when reviewing tasks involving sensitive or personal data
- Work inside isolated test environments
- Verify environment resets between test runs
- Flag and escalate safety-relevant or ambiguous findings
Required qualifications
- Bachelor's degree or equivalent practical experience
- 2–5+ years in quality review, QA/QC, trust & safety, content moderation, data annotation, or a similarly judgment-intensive review role
- Strong written communication
- Comfort with ambiguity and structured decision-making under a rubric
Nice to have
- Baseline understanding of AI/ML concepts and how AI agents complete tasks
About the company
Innodata is a global data engineering company. They provide data, evaluation frameworks, and human expertise to build AI systems.