Senior Data Labeling Specialist — Remote AI QA & RLHF

Rex.zone; United States; On-site; USD 90,000 - 120,000; Full time

Job summary

Rex.zone is hiring a Senior Data Labeling Specialist to deliver high-accuracy annotations and evaluations used to train AI systems. This role is remote for US candidates, with Toronto as the hiring focus. You will work across text, image, and multimodal data to support RLHF and model evaluation.

You’ll apply guidelines, perform QA checks, and help calibrate annotators to shared standards while collaborating with data ops and engineering to improve workflows.

Qualifications

• Experience labeling data at scale with attention to training data quality.
• Ability to apply annotation guidelines, perform QA evaluation, and justify decisions.
• Familiarity with RLHF workflows, evaluation rubrics, and feedback loops.
• Strong written English and meticulous, detail-oriented work style.

Responsibilities

• Execute complex data labeling and evaluation tasks across text, image, and multimodal data.
• Create RLHF preference comparisons, write concise rationales, and flag model failure modes.
• Perform prompt evaluation and LLM evaluation using structured rubrics.
• Run QA audits, track inter-annotator agreement, and calibrate annotators to shared standards.
• Maintain annotation guidelines compliance; resolve ambiguous edge cases and document decisions.
• Partner with engineering and data ops to improve workflows, sampling strategies, and feedback loops.

Skills

Data labeling at scale

QA evaluation

RLHF workflows

Written English

Job description

Rex.zone is hiring a Senior Data Labeling Specialist to deliver high-accuracy annotations and evaluations used to train AI systems. This role is remote for US candidates, with Toronto as the hiring focus. You will work across text, image, and multimodal data to support RLHF and model evaluation.

You’ll apply guidelines, perform QA checks, and help calibrate annotators to shared standards while collaborating with data ops and engineering to improve workflows.