Senior AI Data Annotation & RLHF Evaluator (Remote)

Rex.zone; Miami (FL); On-site; USD 55,000 - 85,000; Full time

Job summary

Rex.zone is seeking a Senior AI data annotator to perform data labeling and evaluation for NLP tasks, RLHF assessments, and prompt QA to improve training data quality and model performance. This is a US-based remote, full-time role aligned with Miami talent demand.

You will apply strict rubrics, document edge cases, and collaborate asynchronously with reviewers and leads. A mid-senior background in data annotation and familiarity with RLHF concepts are required; CV annotation experience is a

Qualifications

• Mid-senior experience in data annotation or AI evaluation with strong attention to detail.
• Ability to apply strict rubrics consistently and communicate clearly in writing.
• Familiarity with RLHF concepts, prompt evaluation, QA evaluation, and training data quality practices.
• Experience with LLM evaluation rubrics, policy-based content labeling, and dataset curation for instruction tuning.
• Exposure to inter-annotator agreement, calibration sessions, and ambiguity resolution.
• Computer vision annotation experience (bounding boxes, segmentation, attribute tagging) is a plus.

Responsibilities

• Perform data labeling for NLP tasks (e.g., NER, classification, summarization quality checks, instruction-following evaluation).
• Conduct RLHF evaluations including pairwise ranking, preference labeling, and writing clear feedback rationales.
• Execute prompt evaluation and QA evaluation for helpfulness, correctness, and policy adherence.
• Run content safety labeling for sensitive content, toxicity, and policy violations.
• Audit training data quality to reduce label noise; flag duplicates and leakage risks.
• Document edge cases, propose rubric improvements, and support dataset curation for LLM training pipelines.
• Collaborate asynchronously with reviewers, QA, and project leads; participate in calibration sessions.

Skills

Data annotation

AI evaluation

Attention to detail

Written communication

RLHF concepts

Prompt evaluation

Job description

Rex.zone is seeking a Senior AI data annotator to perform data labeling and evaluation for NLP tasks, RLHF assessments, and prompt QA to improve training data quality and model performance. This is a US-based remote, full-time role aligned with Miami talent demand.

You will apply strict rubrics, document edge cases, and collaborate asynchronously with reviewers and leads. A mid-senior background in data annotation and familiarity with RLHF concepts are required; CV annotation experience is a