Remote Data Annotator Jobs Toronto

Rex.zone; Toronto; Remote; CAD 42,000 - 62,000; Full time

Job summary

Rex.zone is hiring Toronto-based candidates for full-time remote data annotation and labeling work to improve AI/ML training data quality. You will support real-world LLM training pipelines through evaluation, QA, and rubric-driven judgments.

As a Remote Data Annotator, you will create and evaluate labeled datasets for NLP, computer vision, and content safety labeling, following strict guidelines and documenting edge cases to reduce label noise.

Qualifications

• Mid-Senior experience in data annotation, labeling, QA evaluation, or LLM evaluation.

Responsibilities

• Produce high-accuracy data labeling for text, image, and mixed-modality data.
• Execute RLHF comparisons, preference judgments, and rationale capture.
• Perform prompt evaluation and rubric grading for helpfulness, correctness, and policy adherence.
• Complete NLP annotations such as named entity recognition, classification, and entity linking.
• Support computer vision annotation including bounding boxes, polygons, segmentation masks, and attribute tagging.
• Conduct content safety labeling across harassment, self-harm, sexual content, violence, and sensitive traits.
• Run QA evaluation using gold sets, spot checks, reviewer audits, inter-annotator agreement, and defect taxonomy.

Skills

Data annotation

QA evaluation

Rubric interpretation

Written communication

Remote work reliability

Job description

About Rex.zone

Rex.zone is hiring Toronto-based candidates for full-time remote data annotation and data labeling work that improves training data quality for AI/ML systems. You will support real-world LLM training pipelines through evaluation, QA, and careful rubric-driven judgments.

About Rex.zone

Rex.zone is hiring Toronto-based candidates for full-time remote data annotation and data labeling work that improves training data quality for AI/ML systems. You will support real-world LLM training pipelines through evaluation, QA, and careful rubric-driven judgments.

About The Role

As a Remote Data Annotator, you will create and evaluate labeled datasets used in large language model evaluation, RLHF workflows, NLP tasks (e.g., named entity recognition), computer vision annotation, and content safety labeling. You will follow strict annotation guidelines compliance, document edge cases, and collaborate asynchronously to improve model performance outcomes.

Key Responsibilities

• Produce high-accuracy data labeling for text, image, and mixed-modality tasks
• Execute RLHF comparisons, preference judgments, response scoring, and rationale capture
• Perform prompt evaluation and rubric grading for helpfulness, correctness, and policy adherence
• Complete NLP annotations such as named entity recognition, classification, and entity linking using defined ontologies
• Support computer vision annotation including bounding boxes, polygons, segmentation masks, and attribute tagging
• Conduct content safety labeling across harassment, self-harm, sexual content, violence, and sensitive traits
• Run QA evaluation using gold sets, spot checks, reviewer audits, inter-annotator agreement, and defect taxonomies
• Report ambiguous examples, escape edge cases, and propose guideline clarifications to reduce label noise

Required Qualifications

• Mid-Senior experience in data annotation, data labeling, QA evaluation, or LLM evaluation
• Ability to interpret detailed rubrics and maintain consistent decision-making
• Strong written communication for edge-case documentation and rationale writing
• Comfort working with structured taxonomies (NER, content safety, prompt evaluation)
• Reliability in meeting throughput and quality targets in a remote environment

Tools & Quality Standards

You will work in web-based labeling platforms and evaluation consoles using versioned guidelines and task queues. Quality is measured through sampling, consensus review, inter-annotator agreement checks, and defect tagging. You must be able to handle potentially sensitive content and follow confidentiality requirements.