Senior Data Annotation & AI QA Lead (Remote, US)

Rex.zone

Austin (TX)

Remote

USD 41,328 - 68,880

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Rex.zone is seeking a Senior Data Annotation professional for an Austin-aligned, fully remote role within the US. You will lead end-to-end data annotation and evaluation workflows that support AI/ML training pipelines, including RLHF, prompt-response evaluation, and rubric-based scoring for large language model evaluation.

The role requires experience in data annotation, RLHF familiarity, strong attention to detail, and ability to document edge cases clearly.

Qualifications

  • Experience in data annotation, data labeling, or QA evaluation for AI/ML systems.
  • Familiarity with RLHF and large language model evaluation methodologies.
  • Able to follow and improve annotation guidelines under ambiguity.
  • Strong attention to detail and rubric-based judgment.
  • Experience with NLP tasks such as classification and named entity recognition.
  • Professional communication for documenting edge cases and labeling decisions.

Responsibilities

  • Execute and review data labeling for NLP and (as needed) computer vision datasets
  • Perform RLHF evaluations (preference ranking, pairwise comparisons, structured scoring)
  • Run QA evaluation audits for accuracy, consistency, and policy alignment
  • Conduct prompt evaluation for helpfulness, correctness, and safety
  • Apply named entity recognition tagging and taxonomy mapping where required
  • Support content safety labeling using policy-based classification
  • Write/refine annotation guidelines, edge-case notes, and decision logs
  • Track dataset health metrics (agreement rate, defect density, rework rate) and propose fixes
  • Collaborate with engineering on tooling feedback (labeling UI, workflow automation, sampling strategy)

Skills

Data annotation
RLHF
NLP tagging
Quality evaluation
Documentation

Job description

Rex.zone is seeking a Senior Data Annotation professional for an Austin-aligned, fully remote role within the US. You will lead end-to-end data annotation and evaluation workflows that support AI/ML training pipelines, including RLHF, prompt-response evaluation, and rubric-based scoring for large language model evaluation.

The role requires experience in data annotation, RLHF familiarity, strong attention to detail, and ability to document edge cases clearly.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Annotator: AI Evaluation & QA Lead (Remote)
Senior Data Annotator: AI Evaluation & QA Lead (Remote)

Rex.zone • Phoenix (AZ)

Remote
USD 41,328 - 68,880
Senior AI Data Annotation & Evaluation Specialist (Remote)
Senior AI Data Annotation & Evaluation Specialist (Remote)

Rex.zone • Seattle (WA)

Remote
USD 41,328 - 68,880
Remote Senior Data Annotation Lead for AI Training & QA
Remote Senior Data Annotation Lead for AI Training & QA

Rex.zone • Denver (CO)

Remote
USD 41,328 - 68,880
Senior Data Annotation & Evaluation Lead (Remote)
Senior Data Annotation & Evaluation Lead (Remote)

Rex.zone • Dallas (TX)

Remote
USD 41,328 - 68,880
Remote work
Remote Senior AI Data Annotation Specialist — RLHF & CV QA
Remote Senior AI Data Annotation Specialist — RLHF & CV QA

Rex.zone • Atlanta (GA)

Remote
USD 41,328 - 68,880
Senior AI Data Annotation & RLHF Evaluator (Remote)
Senior AI Data Annotation & RLHF Evaluator (Remote)

Rex.zone • Miami (FL)

Remote
USD 55,000 - 85,000
Remote Senior AI Data Labeler - RLHF, NER & CV Annotation
Remote Senior AI Data Labeler - RLHF, NER & CV Annotation

Rex.zone • Austin (TX)

Remote
USD 65,000 - 95,000
Senior AI Data Annotation Lead (Remote | Full-Time)
Senior AI Data Annotation Lead (Remote | Full-Time)

Rex.zone • United States

Remote
USD 60,000 - 90,000
Senior Data Annotation & RLHF QA Specialist
Senior Data Annotation & RLHF QA Specialist

Rex.zone • Boston (MA)

Remote
USD 70,000 - 110,000
Senior Data Annotation Lead - Remote AI/ML Evaluation
Senior Data Annotation Lead - Remote AI/ML Evaluation

Rex.zone • Miami (FL)

Remote
USD 41,328 - 68,880