Remote Data Annotator

RemoExperts

Montreal

Remote

CAD 59,000 - 98,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Rex.zone is seeking a Remote Data Annotator to join full-time, fully remote work from anywhere. You will annotate text, images, audio, and multi-turn conversations according to detailed guidelines to support AI/ML training pipelines.

The role emphasizes high-quality critique, rubric-based scoring, and RLHF-style feedback, with a base hourly rate of $30–$50 USD and a flexible remote setup. Experience with NLP/LLM evaluation is preferred.

Qualifications

  • Mid-to-senior experience in data annotation, data labeling, or QA evaluation.
  • Strong guideline compliance, consistency, and written reasoning for critique-based tasks.
  • Comfort with structured labeling tools, taxonomies, and metrics-driven remote work.
  • Familiarity with NLP/LLM evaluation or content moderation taxonomies.

Responsibilities

  • Apply detailed annotation guidelines to label and review text, images, audio, and multi-turn conversations.
  • Perform LLM evaluation and prompt evaluation (helpfulness, harmlessness, factuality, reasoning quality).
  • Complete RLHF-style preference labeling, rubric scoring, and critique writing to generate high-signal feedback.
  • Run QA evaluation using sampling plans, inter-annotator agreement checks, and corrective actions.
  • Document edge cases, ambiguity, and rationale to improve rubrics, taxonomies, and consistency.
  • Flag dataset issues (duplicates, leakage, label noise) and follow confidentiality/data-handling standards.

Skills

Data annotation
Guideline compliance
Written reasoning
Remote work familiarity

Tools

Structured labeling tools

Job description

Remote Data Annotator (Full-Time, Remote)

Rex.zone helps candidates discover and apply to remote data annotator roles supporting modern AI/ML training pipelines. In this role, you will create labeled datasets and evaluation signals that improve training data quality and large language model performance across NLP, computer vision, and content safety workflows.

What You Will Do
  • Apply detailed annotation guidelines to label and review text, images, audio, and multi-turn conversations
  • Perform LLM evaluation and prompt evaluation (helpfulness, harmlessness, factuality, reasoning quality)
  • Complete RLHF-style preference labeling, rubric scoring, and critique writing to generate high-signal feedback
  • Run QA evaluation using sampling plans, inter-annotator agreement checks, and corrective actions
  • Document edge cases, ambiguity, and rationale to improve rubrics, taxonomies, and consistency
  • Flag dataset issues (duplicates, leakage, label noise) and follow confidentiality/data-handling standards
Required Qualifications
  • Mid-to-senior experience in data annotation, data labeling, or QA evaluation
  • Strong guideline compliance, consistency, and written reasoning for critique-based tasks
  • Comfort with structured labeling tools, taxonomies, and metrics-driven remote work
  • Familiarity with NLP/LLM evaluation or content moderation taxonomies
Preferred
  • RLHF workflows, preference ranking, and rubric-based scoring experience
  • Computer vision annotation exposure (bounding boxes, polygons, segmentation QA)
  • Understanding of training data quality concepts (bias, representativeness, label noise)
Compensation

Base hourly rate: $30-$50/hr (USD), depending on project needs and qualifications.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote Data Annotator Jobs Toronto
Remote Data Annotator Jobs Toronto

Rex.zone • Toronto

Remote
CAD 42,000 - 62,000
Remote Data Labeling Jobs in Ottawa
Remote Data Labeling Jobs in Ottawa

Rex.zone • Ottawa

Remote
CAD 59,000 - 98,000
Remote Data Labeling Specialist (Canada)
Remote Data Labeling Specialist (Canada)

Rex.zone • Canada

Remote
CAD 48,000 - 68,000
Remote Data Labeling Jobs in Ottawa
Remote Data Labeling Jobs in Ottawa

RemoExperts • Ottawa

Hybrid
CAD 59,000 - 98,000
Senior Data Annotation & QA Lead (Remote, Full-Time)
Senior Data Annotation & QA Lead (Remote, Full-Time)

Rex.zone • Montreal

Remote
CAD 60,000 - 90,000
Remote Data Labeling Specialist for AI Training QA
Remote Data Labeling Specialist for AI Training QA

Rex.zone • Canada

Remote
CAD 48,000 - 68,000
Annotation Quality Editor (Remote)
Annotation Quality Editor (Remote)

Work Vista • Canada

Remote
CAD 28,474 - 34,169
Flexible hours
100% remote work
Opportunity to work up to 40 hours weekly
Robotics Data Annotation Specialist
Robotics Data Annotation Specialist

RemoExperts • Canada

Remote
CAD 60,000 - 90,000
English Language Specialist
English Language Specialist

iMerit Scholars • Canada

Remote
CAD 50,000 - 70,000
Competitive compensation
Performance-based incentives
Flexible working conditions
Remote AI Training Data Annotator (US)
Remote AI Training Data Annotator (US)

Remote Talent Cloud • New Brunswick

Remote
USD 23,000 - 32,000
Fully remote
Full-time & Part-time
Competitive hourly pay
+2