Senior Data Annotator: AI Evaluation & QA Lead (Remote)

Rex.zone; Phoenix (AZ); On-site; USD 41,328 - 68,880; Full time

Job summary

Rex.zone is seeking a Senior Data Annotator to work remotely with our US-based Phoenix team. You will lead data labeling across NLP, LLM evaluation, and computer vision tasks, guiding RLHF workflows and QA checks to ensure high-quality training data.

Sources emphasize strong attention to rubric-based scoring, policy application, and clear edge-case communication. This full-time, remote role offers collaboration with distributed engineers and data ops while focusing on risk-aware content labeling.

Qualifications

• Mid-Senior experience in data annotation, data labeling, or AI evaluation.
• Strong reading comprehension and attention to detail for rubric-based scoring and policy application.
• Familiarity with RLHF task formats, LLM evaluation concepts, and training data quality practices.
• Ability to communicate edge cases clearly and improve annotation guidelines compliance.
• Comfort working remotely with distributed teams on a full-time schedule.

Responsibilities

• Perform senior-level data labeling across NLP, LLM evaluation, and computer vision annotation tasks.
• Execute RLHF workflows such as preference ranking, rubric-based scoring, and pairwise comparisons.
• Conduct QA evaluation: spot checks, gold-set validation, and inter-annotator agreement reviews.
• Apply annotation guidelines compliance and document edge-case decisions for repeatability.
• Run error analysis to identify systematic labeling issues impacting training data quality.
• Provide feedback to improve taxonomy, rubrics, and evaluation criteria.
• Partner with engineering/data ops on task instructions, sampling, and dataset versioning.
• Support content safety labeling aligned to policy and risk categories.

Skills

Data annotation

RLHF

LLM evaluation

Policy application

Remote work

Job description

Rex.zone is seeking a Senior Data Annotator to work remotely with our US-based Phoenix team. You will lead data labeling across NLP, LLM evaluation, and computer vision tasks, guiding RLHF workflows and QA checks to ensure high-quality training data.

Sources emphasize strong attention to rubric-based scoring, policy application, and clear edge-case communication. This full-time, remote role offers collaboration with distributed engineers and data ops while focusing on risk-aware content labeling.