Senior Data Annotation & QA Lead (Remote, Full-Time)

Rex.zone; Montreal (administrative region); On-site; CAD 60,000 - 90,000; Full time

Job summary

Rex.zone in Montreal invites experienced data annotation professionals to join as a senior data annotation specialist. You will support LLM training pipelines through labeling, RLHF evaluation, prompt evaluation, and QA review in a remote, full-time capacity.

The role emphasizes high-quality data, taxonomy design, and rigorous QA processes while collaborating with ML engineers and researchers to improve model performance and safety. Remote work is required and full-time commitment is expected.

Qualifications

• Senior data annotation operations experience with QA ownership is required.
• Strong understanding of taxonomy design, guideline adherence, and ambiguity resolution is required.
• Hands-on experience with LLM evaluation, RLHF, or prompt evaluation workflows is preferred.
• Familiarity with NLP tasks such as named entity recognition and text classification is beneficial.
• Comfortable working remotely with cross-functional stakeholders.

Responsibilities

• Deliver high-accuracy data labeling and review across NLP, LLM evaluation, computer vision annotation, and content safety labeling.
• Design, maintain, and enforce annotation guidelines compliance and taxonomy standards.
• Lead QA evaluation workflows including sampling plans, inter-annotator agreement checks, and adjudication.
• Execute RLHF tasks such as pairwise ranking, preference labeling, and consistency audits.
• Conduct prompt evaluation and rubric-based scoring aligned to model requirements.
• Perform dataset audits, ambiguity logging, and label error analysis to improve training data quality.
• Partner with engineering/ML stakeholders to define acceptance criteria and feedback loops into LLM training pipelines.
• Document edge cases and escalation paths to reduce guideline drift.
• Support onboarding and calibration using examples, gold sets, and retraining plans.
• Track quality metrics and recommend process improvements that improve throughput without sacrificing accuracy.

Skills

QA evaluation ownership

Taxonomy design

Ambiguity resolution

LLM evaluation

NLP tasks

Remote collaboration

Job description

Rex.zone in Montreal invites experienced data annotation professionals to join as a senior data annotation specialist. You will support LLM training pipelines through labeling, RLHF evaluation, prompt evaluation, and QA review in a remote, full-time capacity.

The role emphasizes high-quality data, taxonomy design, and rigorous QA processes while collaborating with ML engineers and researchers to improve model performance and safety. Remote work is required and full-time commitment is expected.