Senior Data Annotation & Evaluation Lead (Remote)

Rex.zone

Dallas (TX)

Remote

USD 41,328 - 68,880

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Remote work

Job summary

Rex.zone is seeking a Senior Data Annotation Specialist to lead high-quality labeling and evaluation workflows that enhance training data quality and LLM evaluation. This full-time remote role is aligned with Dallas-area recruiting needs and US-based projects.

You will execute NLP, NER, and document understanding labeling, perform RLHF-style ranking, and contribute to prompt evaluation and QA. Strong guideline enforcement and collaboration with AI/ML teams are essential.

Qualifications

  • Experience following and enforcing detailed annotation guidelines in edge cases.
  • Experience with training data quality processes (sampling, adjudication, calibration, error taxonomy).
  • Exposure to LLM evaluation, RLHF-style ranking, or rubric-based prompt evaluation.

Responsibilities

  • Execute and review data labeling tasks for NLP, NER, and document understanding.
  • Perform RLHF evaluation and comparative ranking to improve alignment and helpfulness.
  • Conduct prompt evaluation using calibrated rubrics for correctness and policy compliance.
  • Run QA evaluation to ensure consistency and high training data quality.
  • Create and refine annotation guidelines and decision trees to maintain compliance.
  • Support computer vision annotation (boxes, polygons, keypoints) when needed.
  • Label and review content safety data across policy categories with high precision.
  • Document dataset issues and propose fixes to improve model performance.
  • Collaborate with AI/ML and data ops teams to improve quality metrics in pipelines.

Skills

Data annotation
RLHF evaluation
Guideline enforcement
NLP labeling
Quality assurance
Remote collaboration

Job description

Rex.zone is seeking a Senior Data Annotation Specialist to lead high-quality labeling and evaluation workflows that enhance training data quality and LLM evaluation. This full-time remote role is aligned with Dallas-area recruiting needs and US-based projects.

You will execute NLP, NER, and document understanding labeling, perform RLHF-style ranking, and contribute to prompt evaluation and QA. Strong guideline enforcement and collaboration with AI/ML teams are essential.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Annotator: AI Evaluation & QA Lead (Remote)
Senior Data Annotator: AI Evaluation & QA Lead (Remote)

Rex.zone • Phoenix (AZ)

Remote
USD 41,328 - 68,880
Senior Data Annotation Lead - Remote AI/ML Evaluation
Senior Data Annotation Lead - Remote AI/ML Evaluation

Rex.zone • Miami (FL)

Remote
USD 41,328 - 68,880
Senior AI Data Annotation Lead (Remote | Full-Time)
Senior AI Data Annotation Lead (Remote | Full-Time)

Rex.zone • United States

Remote
USD 60,000 - 90,000
Remote Senior Data Annotation Lead for AI Training & QA
Remote Senior Data Annotation Lead for AI Training & QA

Rex.zone • Denver (CO)

Remote
USD 41,328 - 68,880
Senior AI Data Annotation & Evaluation Specialist (Remote)
Senior AI Data Annotation & Evaluation Specialist (Remote)

Rex.zone • Seattle (WA)

Remote
USD 41,328 - 68,880
Senior Data Annotation Lead - Remote NLP, RLHF & CV
Senior Data Annotation Lead - Remote NLP, RLHF & CV

Rex.zone • Los Angeles (CA)

Remote
USD 80,000 - 120,000
Senior Data Annotator - Remote AI Training & Evaluation
Senior Data Annotator - Remote AI Training & Evaluation

Rex.zone • Seattle (WA)

Remote
USD 41,328 - 68,880
Remote AI Data Annotation & Evaluation Specialist
Remote AI Data Annotation & Evaluation Specialist

Rex.zone • Detroit (MI)

Remote
USD 41,328 - 68,880
Senior Data Labeling Specialist (Remote) — AI/LLM QA
Senior Data Labeling Specialist (Remote) — AI/LLM QA

Rex.zone • United States

Remote
USD 85,000 - 130,000
Senior Data Annotation & AI QA Lead (Remote, US)
Senior Data Annotation & AI QA Lead (Remote, US)

Rex.zone • Austin (TX)

Remote
USD 41,328 - 68,880