Senior AI Data Annotation & Evaluation Specialist (Remote)

Rex.zone; Seattle (WA); On-site; USD 41,328 - 68,880; Full time

Job summary

Rex.zone is seeking a remote, full-time Senior AI Data Annotation role supporting training-data quality for modern AI/ML systems. You will deliver high-precision human feedback across RLHF, LLM evaluation, prompt evaluation, and QA evaluation to drive measurable model performance improvement.

You will collaborate with engineers and data ops on task design, schema updates, and quality monitoring signals, documenting edge cases and escalating issues that impact data quality.

Qualifications

• Experience in AI data annotation and/or production LLM evaluation.
• Understanding of RLHF concepts and how human feedback improves model alignment.
• Ability to follow and improve annotation guidelines with high attention to detail.
• Familiarity with QA evaluation methods (audits, gold sets, calibration).
• Experience with NLP labeling (classification, sentiment, NER) and/or CV annotation.
• Excellent written communication for documenting decisions, rationales, and edge cases.

Responsibilities

• Deliver senior-level AI data annotation across text, image, and mixed-modality tasks.
• Execute RLHF workflows including preference ranking, pairwise comparisons, and rationale capture.
• Perform rubric-based LLM evaluation and QA evaluation for helpfulness, honesty, harmlessness, and policy compliance.
• Run prompt evaluation to identify failure modes, jailbreak risk, and instruction-following gaps.
• Apply strict annotation guidelines compliance; report ambiguities and propose guideline improvements.
• Conduct error analysis, label auditing, calibration, and consensus reviews to stabilize quality.
• Support content safety labeling with structured QA review.
• Collaborate with engineers/data ops on task design, schema updates, and quality monitoring signals.
• Document edge cases clearly and elevate issues that impact training data quality.

Skills

AI data annotation

RLHF concepts

NLP labeling

Quality auditing

Remote collaboration

Documentation

Job description

Rex.zone is seeking a remote, full-time Senior AI Data Annotation role supporting training-data quality for modern AI/ML systems. You will deliver high-precision human feedback across RLHF, LLM evaluation, prompt evaluation, and QA evaluation to drive measurable model performance improvement.

You will collaborate with engineers and data ops on task design, schema updates, and quality monitoring signals, documenting edge cases and escalating issues that impact data quality.