Remote Data Annotator

Rex.zone; Montreal (administrative region); Remote; CAD 59,000 - 98,000; Full time

Job summary

Rex.zone is seeking a Remote Data Annotator to join full-time, fully remote work from anywhere. You will annotate text, images, audio, and multi-turn conversations according to detailed guidelines to support AI/ML training pipelines.

The role emphasizes high-quality critique, rubric-based scoring, and RLHF-style feedback, with a base hourly rate of $30–$50 USD and a flexible remote setup. Experience with NLP/LLM evaluation is preferred.

Qualifications

• Mid-to-senior experience in data annotation, data labeling, or QA evaluation.
• Strong guideline compliance, consistency, and written reasoning for critique-based tasks.
• Comfort with structured labeling tools, taxonomies, and metrics-driven remote work.
• Familiarity with NLP/LLM evaluation or content moderation taxonomies.

Responsibilities

• Apply detailed annotation guidelines to label and review text, images, audio, and multi-turn conversations.
• Perform LLM evaluation and prompt evaluation (helpfulness, harmlessness, factuality, reasoning quality).
• Complete RLHF-style preference labeling, rubric scoring, and critique writing to generate high-signal feedback.
• Run QA evaluation using sampling plans, inter-annotator agreement checks, and corrective actions.
• Document edge cases, ambiguity, and rationale to improve rubrics, taxonomies, and consistency.
• Flag dataset issues (duplicates, leakage, label noise) and follow confidentiality/data-handling standards.

Skills

Data annotation

Guideline compliance

Written reasoning

Remote work familiarity

Tools

Structured labeling tools

Job description

Remote Data Annotator (Full-Time, Remote)

Rex.zone helps candidates discover and apply to remote data annotator roles supporting modern AI/ML training pipelines. In this role, you will create labeled datasets and evaluation signals that improve training data quality and large language model performance across NLP, computer vision, and content safety workflows.

What You Will Do

• Apply detailed annotation guidelines to label and review text, images, audio, and multi-turn conversations
• Perform LLM evaluation and prompt evaluation (helpfulness, harmlessness, factuality, reasoning quality)
• Complete RLHF-style preference labeling, rubric scoring, and critique writing to generate high-signal feedback
• Run QA evaluation using sampling plans, inter-annotator agreement checks, and corrective actions
• Document edge cases, ambiguity, and rationale to improve rubrics, taxonomies, and consistency
• Flag dataset issues (duplicates, leakage, label noise) and follow confidentiality/data-handling standards

Required Qualifications

• Mid-to-senior experience in data annotation, data labeling, or QA evaluation
• Strong guideline compliance, consistency, and written reasoning for critique-based tasks
• Comfort with structured labeling tools, taxonomies, and metrics-driven remote work
• Familiarity with NLP/LLM evaluation or content moderation taxonomies

Preferred

• RLHF workflows, preference ranking, and rubric-based scoring experience
• Computer vision annotation exposure (bounding boxes, polygons, segmentation QA)
• Understanding of training data quality concepts (bias, representativeness, label noise)

Compensation

Base hourly rate: $30-$50/hr (USD), depending on project needs and qualifications.