Senior Data Annotation & AI QA Lead (Remote, US)

Rex.zone; Austin (TX); On-site; USD 41,328 - 68,880; Full time

Job summary

Rex.zone is seeking a Senior Data Annotation professional for an Austin-aligned, fully remote role within the US. You will lead end-to-end data annotation and evaluation workflows that support AI/ML training pipelines, including RLHF, prompt-response evaluation, and rubric-based scoring for large language model evaluation.

The role requires experience in data annotation, RLHF familiarity, strong attention to detail, and ability to document edge cases clearly.

Qualifications

• Experience in data annotation, data labeling, or QA evaluation for AI/ML systems.
• Familiarity with RLHF and large language model evaluation methodologies.
• Able to follow and improve annotation guidelines under ambiguity.
• Strong attention to detail and rubric-based judgment.
• Experience with NLP tasks such as classification and named entity recognition.
• Professional communication for documenting edge cases and labeling decisions.

Responsibilities

• Execute and review data labeling for NLP and (as needed) computer vision datasets
• Perform RLHF evaluations (preference ranking, pairwise comparisons, structured scoring)
• Run QA evaluation audits for accuracy, consistency, and policy alignment
• Conduct prompt evaluation for helpfulness, correctness, and safety
• Apply named entity recognition tagging and taxonomy mapping where required
• Support content safety labeling using policy-based classification
• Write/refine annotation guidelines, edge-case notes, and decision logs
• Track dataset health metrics (agreement rate, defect density, rework rate) and propose fixes
• Collaborate with engineering on tooling feedback (labeling UI, workflow automation, sampling strategy)

Skills

Data annotation

RLHF

NLP tagging

Quality evaluation

Documentation

Job description

Rex.zone is seeking a Senior Data Annotation professional for an Austin-aligned, fully remote role within the US. You will lead end-to-end data annotation and evaluation workflows that support AI/ML training pipelines, including RLHF, prompt-response evaluation, and rubric-based scoring for large language model evaluation.

The role requires experience in data annotation, RLHF familiarity, strong attention to detail, and ability to document edge cases clearly.