Remote AI Training Data QA Annotator

Rex.zone; United States; Remote; USD 41,000 - 69,000; Full time

Job summary

Rex.zone is seeking mid-senior data annotators to support AI training workflows, including data labeling, RLHF evaluation, prompt evaluation, and training data QA. This remote role follows structured rubrics, resolves edge cases, and requires delivering high-quality labeled datasets that improve model performance across real-world LLM training pipelines.

Responsibilities include labeling text, image, and safety data, performing RLHF comparisons, and contributing to calibration sessions to

Qualifications

• Mid-senior experience in data annotation, QA evaluation, or AI data operations.
• Strong written English and high attention to detail.
• Proven ability to follow rubrics and maintain annotation guidelines compliance.
• Familiarity with NLP concepts (classification, NER) and/or computer vision annotation.
• Comfort working remotely with feedback cycles, quality audits, and production tooling.

Responsibilities

• Deliver accurate labels and evaluations across text, image, and safety datasets.
• Perform RLHF preference ranking (pairwise comparisons) and preference judgments.
• Conduct prompt response evaluation for helpfulness, factuality, and policy compliance.
• Complete named entity recognition (NER) and intent/classification tasks.
• Annotate computer vision data (e.g., bounding boxes, polygons, segmentation) as required.
• Execute rubric-based QA evaluation; elevate ambiguous edge cases with clear rationales.
• Participate in calibration sessions to reduce inter‑annotator variance and improve agreement.
• Document decisions to improve annotation guidelines and audit readiness.
• Meet throughput targets while maintaining consistency, precision, and quality.

Skills

Data annotation

QA evaluation

AI data operations

Job description

Rex.zone is seeking mid-senior data annotators to support AI training workflows, including data labeling, RLHF evaluation, prompt evaluation, and training data QA. This remote role follows structured rubrics, resolves edge cases, and requires delivering high-quality labeled datasets that improve model performance across real-world LLM training pipelines.

Responsibilities include labeling text, image, and safety data, performing RLHF comparisons, and contributing to calibration sessions to