Senior Data Labeling Lead — Remote AI Evaluation & QA

Rex.zone; Chicago (IL); On-site; USD 41,000 - 69,000; Full time

Job summary

Rex.zone is seeking senior data labeling professionals to drive high-quality labeled datasets and evaluation signals that enhance AI/ML system performance. This remote role focuses on NLP, LLM evaluation, and computer vision annotation for complex projects across diverse datasets.

You will lead labeling tasks, implement RLHF, and perform rigorous QA with gold sets and adjudication, while documenting decisions and updating guidelines for ongoing quality improvements.

Qualifications

• Mid-senior experience in data labeling/data annotation/AI evaluation
• Strong annotation guidelines compliance and calibrated decision‑making under ambiguity
• Hands‑on QA practices (gold sets, inter‑annotator agreement, adjudication)
• Familiarity with RLHF, preference labeling, and prompt evaluation for LLMs
• Excellent written communication for rationales and decision logs

Responsibilities

• Lead complex NLP, LLM evaluation, and computer vision annotation projects
• Execute RLHF preference labeling (pairwise preference, ranking, justifications) and prompt evaluation
• Run QA evaluation using gold sets, sampling plans, spot checks, and adjudication
• Perform named entity recognition and structured extraction labeling with consistent schemas
• Label content safety categories (hate, harassment, self-harm, sexual content, violence) using policy rubrics
• Analyze disagreement patterns, create error taxonomies, and drive training data quality improvements
• Partner with engineering/product on dataset specs, schema, and acceptance criteria
• Maintain documentation, decision logs, and versioned guideline updates; mentor annotators

Skills

Data labeling

QA practices

Communication

Job description

Rex.zone is seeking senior data labeling professionals to drive high-quality labeled datasets and evaluation signals that enhance AI/ML system performance. This remote role focuses on NLP, LLM evaluation, and computer vision annotation for complex projects across diverse datasets.

You will lead labeling tasks, implement RLHF, and perform rigorous QA with gold sets and adjudication, while documenting decisions and updating guidelines for ongoing quality improvements.