Remote Senior Data Annotation Lead for AI Training & QA

Rex.zone; Denver (CO); On-site; USD 41,328 - 68,880; Full time

Job summary

Rex.zone is seeking a Senior data annotation professional to lead complex labeling and evaluation workstreams in a Denver-anchored, remote role. You will shape annotation guidelines, resolve edge cases, and deliver audit-ready datasets that boost AI model performance while ensuring safe deployment and high data quality.

The position emphasizes RLHF, NER and taxonomy classification, and multimodal annotation with strong ownership of QA outcomes.

Qualifications

• Senior or lead annotator with strong quality ownership.
• Familiar with production QA processes, disagreement analysis, and dataset versioning.
• Familiarity with LLM evaluation concepts (preference labels, rubric scoring, prompt evaluation).

Responsibilities

• Execute and review high-difficulty labeling tasks across NLP and multimodal projects.
• Run QA evaluation audits, identify disagreement patterns, and propose guideline updates.
• Perform prompt evaluation and RLHF preference ranking with clear rationales.
• Create gold sets, measure training data quality, and validate dataset integrity before release.
• Lead calibration and quality reviews; ensure annotation guidelines compliance.

Skills

Senior annotator

Quality ownership

QA processes

LLM evaluation

Disagreement analysis

Job description

Rex.zone is seeking a Senior data annotation professional to lead complex labeling and evaluation workstreams in a Denver-anchored, remote role. You will shape annotation guidelines, resolve edge cases, and deliver audit-ready datasets that boost AI model performance while ensuring safe deployment and high data quality.

The position emphasizes RLHF, NER and taxonomy classification, and multimodal annotation with strong ownership of QA outcomes.