Senior Data Annotation & QA Lead (Remote, Full-Time)

Rex.zone

Montreal

Remote

CAD 60,000 - 90,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Rex.zone in Montreal invites experienced data annotation professionals to join as a senior data annotation specialist. You will support LLM training pipelines through labeling, RLHF evaluation, prompt evaluation, and QA review in a remote, full-time capacity.

The role emphasizes high-quality data, taxonomy design, and rigorous QA processes while collaborating with ML engineers and researchers to improve model performance and safety. Remote work is required and full-time commitment is expected.

Qualifications

  • Senior data annotation operations experience with QA ownership is required.
  • Strong understanding of taxonomy design, guideline adherence, and ambiguity resolution is required.
  • Hands-on experience with LLM evaluation, RLHF, or prompt evaluation workflows is preferred.
  • Familiarity with NLP tasks such as named entity recognition and text classification is beneficial.
  • Comfortable working remotely with cross-functional stakeholders.

Responsibilities

  • Deliver high-accuracy data labeling and review across NLP, LLM evaluation, computer vision annotation, and content safety labeling.
  • Design, maintain, and enforce annotation guidelines compliance and taxonomy standards.
  • Lead QA evaluation workflows including sampling plans, inter-annotator agreement checks, and adjudication.
  • Execute RLHF tasks such as pairwise ranking, preference labeling, and consistency audits.
  • Conduct prompt evaluation and rubric-based scoring aligned to model requirements.
  • Perform dataset audits, ambiguity logging, and label error analysis to improve training data quality.
  • Partner with engineering/ML stakeholders to define acceptance criteria and feedback loops into LLM training pipelines.
  • Document edge cases and escalation paths to reduce guideline drift.
  • Support onboarding and calibration using examples, gold sets, and retraining plans.
  • Track quality metrics and recommend process improvements that improve throughput without sacrificing accuracy.

Skills

QA evaluation ownership
Taxonomy design
Ambiguity resolution
LLM evaluation
NLP tasks
Remote collaboration

Job description

Overview

Senior data annotation jobs in Montreal focus on creating and evaluating high-quality training data for AI/ML systems. At Rex.zone, you will support LLM training pipelines through data labeling, RLHF evaluation, prompt evaluation, and QA review to improve model performance and safety in a remote, full-time role.

Overview

Senior data annotation jobs in Montreal focus on creating and evaluating high-quality training data for AI/ML systems. At Rex.zone, you will support LLM training pipelines through data labeling, RLHF evaluation, prompt evaluation, and QA review to improve model performance and safety in a remote, full-time role.

Key Responsibilities

  • Deliver high-accuracy data labeling and review across NLP, LLM evaluation, computer vision annotation, and content safety labeling
  • Design, maintain, and enforce annotation guidelines compliance and taxonomy standards
  • Lead QA evaluation workflows including sampling plans, inter-annotator agreement checks, and adjudication
  • Execute RLHF tasks such as pairwise ranking, preference labeling, and consistency audits
  • Conduct prompt evaluation and rubric-based scoring aligned to model requirements
  • Perform dataset audits, ambiguity logging, and label error analysis to improve training data quality
  • Partner with engineering/ML stakeholders to define acceptance criteria and feedback loops into LLM training pipelines
  • Document edge cases and escalation paths to reduce guideline drift
  • Support onboarding and calibration using examples, gold sets, and retraining plans
  • Track quality metrics and recommend process improvements that improve throughput without sacrificing accuracy

Required Qualifications

  • Senior data annotation or data labeling operations experience with QA evaluation ownership
  • Strong understanding of taxonomy design, guideline adherence, and ambiguity resolution
  • Hands-on experience with LLM evaluation, RLHF, or prompt evaluation workflows
  • Familiarity with NLP tasks such as named entity recognition and text classification
  • Comfortable working remotely with cross-functional stakeholders

Nice to Have

  • Content safety labeling or policy-driven evaluation experience
  • Inter-annotator agreement metrics and quality sampling methodologies
  • Computer vision annotation exposure (bounding boxes, polygons, segmentation)
  • Multilingual evaluation or locale-specific guidelines

Work Model

Remote, Full-time.

Apply with relevant annotation experience and examples of QA evaluation or guideline development work.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote Data Annotator Jobs Toronto
Remote Data Annotator Jobs Toronto

Rex.zone • Toronto

Remote
CAD 42,000 - 62,000
Remote Data Labeling Specialist (Canada)
Remote Data Labeling Specialist (Canada)

Rex.zone • Canada

Remote
CAD 48,000 - 68,000
Remote Data Labeling Jobs in Ottawa
Remote Data Labeling Jobs in Ottawa

Rex.zone • Ottawa

Remote
CAD 59,000 - 98,000
Remote Data Labeling Jobs in Ottawa
Remote Data Labeling Jobs in Ottawa

RemoExperts • Ottawa

Hybrid
CAD 59,000 - 98,000
Remote Data Annotator
Remote Data Annotator

RemoExperts • Montreal

Remote
CAD 59,000 - 98,000
Robotics Data Annotation Specialist
Robotics Data Annotation Specialist

RemoExperts • Canada

Remote
CAD 60,000 - 90,000
Robot Action Annotation Specialist
Robot Action Annotation Specialist

RemoExperts • Canada

Remote
CAD 65,000 - 110,000
English Language Specialist
English Language Specialist

iMerit Scholars • Canada

Remote
CAD 50,000 - 70,000
Competitive compensation
Performance-based incentives
Flexible working conditions
Data Annotation Specialist, Safety
Data Annotation Specialist, Safety

Cohere • Toronto

Remote
CAD 51,000 - 73,000
Data Annotation Specialist, Generalist
Data Annotation Specialist, Generalist

Cohere • Calgary

Remote
CAD 34,095 - 48,560