Cohere

Data Annotation Specialist, Generalist

Canada, Remote. Posted 3 months ago

The role

You will evaluate, rank, and stress-test Large Language Model outputs across text, image, and structured formats. You will also create datasets, build rubrics, and report on model performance to improve model capabilities.

What you will do

  • Evaluate and rank model outputs for accuracy, helpfulness, tone, and safety
  • Stress-test and break models to surface failure modes and unsafe behavior
  • Create datasets by authoring high-quality prompts, responses, and exemplars
  • Build and apply rubrics and taxonomies for grading criteria
  • Annotate and correct multimodal data including text, image, and structured data
  • Participate in calibration exercises and inter-annotator agreement checks
  • Adapt to new and evolving task types as project needs shift
  • Report on model performance and communicate quality trends to partners

Required qualifications

  • 1+ years of experience in AI data annotation, LLM evaluation, content moderation, research, or a related analytical role
  • Experience applying detailed guidelines to complex and ambiguous content
  • Strong contextual and sociocultural judgment
  • Excellent command of written English and strong reading comprehension
  • Strong attention to detail and commitment to accuracy
  • Comfort working with annotation platforms and structured formats such as JSON, CSV/TSV, Markdown, XML, and YAML
  • Ability to work independently in a global, asynchronous team
  • Must use your own device (BYOD)

Nice to have

  • Familiarity with how large language models behave, such as hallucination, sycophancy, and instruction-following gaps
  • Fluency in another language

Before you apply

  • Available within Canada only
  • Role involves exposure to not safe for work (NSFW) text content including explicit or offensive material
  • Initial Screening
  • First Assessment (multiple-choice)
  • Second Assessment (comprehensive)

About the company

Cohere is a security-first enterprise AI company that builds foundation AI models and end-to-end products. They are a global technology company headquartered in Toronto.