Senior Data Annotation & Evaluation Lead (Remote)

Rex.zone; Dallas (TX); On-site; USD 41,328 - 68,880; Full time

Benefits offered by this job

Remote work

Job summary

Rex.zone is seeking a Senior Data Annotation Specialist to lead high-quality labeling and evaluation workflows that enhance training data quality and LLM evaluation. This full-time remote role is aligned with Dallas-area recruiting needs and US-based projects.

You will execute NLP, NER, and document understanding labeling, perform RLHF-style ranking, and contribute to prompt evaluation and QA. Strong guideline enforcement and collaboration with AI/ML teams are essential.

Qualifications

• Experience following and enforcing detailed annotation guidelines in edge cases.
• Experience with training data quality processes (sampling, adjudication, calibration, error taxonomy).
• Exposure to LLM evaluation, RLHF-style ranking, or rubric-based prompt evaluation.

Responsibilities

• Execute and review data labeling tasks for NLP, NER, and document understanding.
• Perform RLHF evaluation and comparative ranking to improve alignment and helpfulness.
• Conduct prompt evaluation using calibrated rubrics for correctness and policy compliance.
• Run QA evaluation to ensure consistency and high training data quality.
• Create and refine annotation guidelines and decision trees to maintain compliance.
• Support computer vision annotation (boxes, polygons, keypoints) when needed.
• Label and review content safety data across policy categories with high precision.
• Document dataset issues and propose fixes to improve model performance.
• Collaborate with AI/ML and data ops teams to improve quality metrics in pipelines.

Skills

Data annotation

RLHF evaluation

Guideline enforcement

NLP labeling

Quality assurance

Remote collaboration

Job description

Rex.zone is seeking a Senior Data Annotation Specialist to lead high-quality labeling and evaluation workflows that enhance training data quality and LLM evaluation. This full-time remote role is aligned with Dallas-area recruiting needs and US-based projects.

You will execute NLP, NER, and document understanding labeling, perform RLHF-style ranking, and contribute to prompt evaluation and QA. Strong guideline enforcement and collaboration with AI/ML teams are essential.