Remote Senior Data Labeling Lead (NLP & CV)

Rex.zone; United States; On-site; USD 90,000 - 130,000; Full time

Job summary

Rex.zone is seeking a Senior Data Labeling Specialist to lead training data quality and evaluation workflows across NLP, computer vision, and LLM training pipelines. This remote role focuses on annotation guidelines compliance, QA evaluation, and reviewer calibration to improve model performance.

You will own taxonomy design, label definitions, and gold-standard examples, and drive RLHF tasks, QA programs, and cross-functional collaboration to deliver scalable labeling quality.

Qualifications

• Mid-Senior experience in data labeling or ML data operations.
• Strong understanding of annotation guidelines and QA methods.
• Hands-on audits, adjudication, and calibration sessions.
• Familiarity with LLM training pipelines, RLHF concepts, preference data, and prompt evaluation best practices.
• Ability to reason through ambiguity and handle long-tail edge cases consistently.
• Comfort collaborating cross-functionally with engineering, applied ML, product, and operations.

Responsibilities

• Own annotation project setup: taxonomy design, label definitions, and gold-standard examples for consistent ground truth creation
• Lead RLHF support tasks including preference ranking and prompt evaluation aligned to safety, helpfulness, and factuality rubrics
• Drive QA evaluation programs: sampling audits, double-pass review, adjudication, error taxonomy, and root-cause analysis
• Support NLP labeling tasks such as named entity recognition (NER), entity linking, classification, and relevance judgments
• Support computer vision annotation workflows (bounding boxes, polygons, segmentation masks, attribute tagging) as needed
• Operationalize content safety labeling guidelines for toxicity, self-harm, policy, and sensitive content categories
• Report training data quality KPIs, inter-annotator agreement, guideline violation rates, and improvement insights to stakeholders
• Coordinate multi-reviewer workflows and collaborate with vendors/BPO teams when applicable
• Continuously iterate guidelines and tooling feedback to reduce ambiguity and improve labeling efficiency

Skills

Data labeling experience

Annotation guidelines

Audits & adjudication

RLHF concepts

Ambiguity handling

Cross-functional collaboration

Job description

Rex.zone is seeking a Senior Data Labeling Specialist to lead training data quality and evaluation workflows across NLP, computer vision, and LLM training pipelines. This remote role focuses on annotation guidelines compliance, QA evaluation, and reviewer calibration to improve model performance.

You will own taxonomy design, label definitions, and gold-standard examples, and drive RLHF tasks, QA programs, and cross-functional collaboration to deliver scalable labeling quality.