Senior Data Labeling Lead - NLP, LLM & Vision

Rex.zone; United States; On-site; USD 60,000 - 90,000; Full time

Job summary

Rex.zone is hiring a senior data labeling specialist to lead day-to-day labeling execution and quality assurance across NLP, LLM evaluation, and computer vision annotation projects. You will apply strong judgment and consistent rubric use to produce reliable training signals that improve model performance.

Join a remote-ready team with responsibilities spanning edge-case resolution, prompt evaluation, and mentorship of labelers.

Qualifications

• Experience in production data labeling/data annotation with QA ownership.
• Familiarity with LLM evaluation, rubric design, and ranking/preference labeling (RLHF workflows).
• Strong written reasoning and attention to detail in long-form prompt evaluation.
• Knowledge of annotation tools, dataset curation, and quality metrics (agreement rates, defect rates).

Responsibilities

• Own training data quality across multiple queues by enforcing annotation guidelines and resolving edge cases.
• Perform RLHF-style ranking, preference labeling, and rubric-based QA evaluation for large language model evaluation.
• Conduct prompt evaluation and response assessment for helpfulness, factuality, toxicity, and policy compliance (content safety labeling).
• Label and review tasks including named entity recognition, text classification, summarization grading, and conversation evaluation.
• Support computer vision annotation (bounding boxes, polygons, keypoints) with strict quality checks when required.
• Run error analysis, identify systematic label noise, and propose guideline updates to reduce ambiguity.
• Mentor labelers, calibrate decision-making, and maintain inter-annotator agreement targets.
• Create and maintain lightweight documentation for task definitions, examples, and escalation paths

Skills

QA ownership

RLHF evaluation

NLP/LLM evaluation

annotation tools

remote work

edge case resolution

Job description

Rex.zone is hiring a senior data labeling specialist to lead day-to-day labeling execution and quality assurance across NLP, LLM evaluation, and computer vision annotation projects. You will apply strong judgment and consistent rubric use to produce reliable training signals that improve model performance.

Join a remote-ready team with responsibilities spanning edge-case resolution, prompt evaluation, and mentorship of labelers.