Senior Data Labeling Specialist (Remote) — AI/LLM QA

Rex.zone; United States; On-site; USD 85,000 - 130,000; Full time

Job summary

Rex.zone is seeking a Senior Data Labeling Specialist to support AI/ML training workflows across NLP and computer vision projects. This is a remote, full-time role delivering high-accuracy RLHF and evaluation datasets and improving training data quality that drives model performance.

You will handle RLHF preference ranking, prompt evaluation, and guideline enforcement, while executing NLP and CV annotation tasks and ensuring auditability of decisions.

Qualifications

• Mid-Senior experience in data labeling, data annotation, or QA evaluation for AI/ML datasets.
• Familiarity with RLHF workflows, prompt evaluation, and LLM evaluation rubrics.
• Strong understanding of annotation guidelines compliance and handling ambiguous edge cases.
• Experience with training data quality methods: sampling, audits, adjudication, and error taxonomies.
• Comfort working in a remote, metric-driven environment (accuracy, throughput, consistency).

Responsibilities

• Perform complex data labeling and data annotation for LLM training pipelines, including RLHF preference ranking and QA evaluation.
• Run prompt evaluation and large language model evaluation using rubrics, scoring scales, and pairwise comparisons.
• Create, refine, and enforce annotation guidelines; adjudicate edge cases and document decisions for auditability.
• Execute NLP tasks such as named entity recognition, text classification, toxicity, factuality checks, and summarization evaluation.
• Execute computer vision annotation tasks (bounding boxes, polygons, keypoints, segmentation masks, attribute labeling).
• Perform content safety labeling (violence, self-harm, hate, sexual content) with consistent taxonomy and escalation pathways.
• Conduct QA: inter-annotator agreement checks, sampling plans, error analysis, audits, and corrective feedback.
• Partner with engineering and operations to improve tooling, reduce label noise, and support model performance improvement.

Skills

Data labeling

QA evaluation

RLHF workflows

LLM evaluation rubrics

Annotation guidelines

Edge-case handling

Job description

Rex.zone is seeking a Senior Data Labeling Specialist to support AI/ML training workflows across NLP and computer vision projects. This is a remote, full-time role delivering high-accuracy RLHF and evaluation datasets and improving training data quality that drives model performance.

You will handle RLHF preference ranking, prompt evaluation, and guideline enforcement, while executing NLP and CV annotation tasks and ensuring auditability of decisions.