Remote Senior AI Data Annotation Specialist — RLHF & CV QA

Rex.zone; Atlanta (GA); On-site; USD 41,328 - 68,880; Full time

Job summary

Rex.zone is hiring a Senior AI Data Annotation Specialist to create, review, and audit high-quality training data for modern AI systems, including LLMs and multimodal models. You will execute annotation tasks, run QA workflows, and support RLHF through ratings and critiques that improve model performance.

The role is remote, full-time, with collaboration across distributed teams in the US. You will work on NLP and computer vision annotation, policy-based evaluation, and data quality audits,

Qualifications

• Experience delivering high-volume, high-precision AI data annotation.
• Strong English comprehension and writing for prompt evaluation and critique.
• Familiarity with RLHF concepts (pairwise ranking, preference data).
• Experience with QA evaluation processes and reviewer calibration.

Responsibilities

• Produce accurate labels while maintaining annotation guidelines compliance.
• Perform QA evaluation, second-pass reviews, and sampling-based audits.
• Calibrate with reviewers to maintain consistency and inter-annotator agreement.
• Document edge cases, resolve disagreements, and propose guideline updates.
• Partner with data operations and engineering to improve tools and workflows and report systematic failure modes observed during evaluations.

Skills

High-volume annotation

RLHF concepts

English writing

QA evaluation

Reviewer calibration

NLP annotation

Computer vision annotation

Job description

Rex.zone is hiring a Senior AI Data Annotation Specialist to create, review, and audit high-quality training data for modern AI systems, including LLMs and multimodal models. You will execute annotation tasks, run QA workflows, and support RLHF through ratings and critiques that improve model performance.

The role is remote, full-time, with collaboration across distributed teams in the US. You will work on NLP and computer vision annotation, policy-based evaluation, and data quality audits,