Remote AI Data Annotation Lead for LLM Training

Rex.zone; Portland (OR); On-site; USD 65,000 - 100,000; Full time

Job summary

Rex.zone is seeking a data annotation professional to lead end-to-end annotation workflows for LLM and multimodal training. You will ensure data quality, drive labeling consistency across projects, and collaborate with engineers and researchers to resolve edge cases that affect model behavior.

You will own dataset-level quality strategy, execute RLHF and LLM evaluation tasks, improve guidelines, and coordinate with stakeholders on data schemas, prompts, and evaluation metrics while supporting

Qualifications

• Experience in AI data annotation or model evaluation programs.
• Skilled in sampling, audits, and data quality measurement.
• Familiar with RLHF, rubric-based evaluation.
• Experience with NLP tasks like NER and taxonomy classification.
• Ability to document decisions and maintain traceability.

Responsibilities

• Own dataset-level quality strategy including gold sets and inter-annotator agreement.
• Execute RLHF and large language model evaluation tasks such as pairwise ranking.
• Perform QA evaluation on labeled data and drive corrective actions.
• Improve annotation guidelines and escalation workflows for edge cases.
• Coordinate with AI/ML engineering stakeholders on data schemas, prompts, and metrics.
• Support content safety labeling and policy-aligned decisions for safety-critical datasets.

Skills

AI data annotation

Sampling & audits

RLHF / LLM eval

NLP tasks

Documentation & traceability

Job description

Rex.zone is seeking a data annotation professional to lead end-to-end annotation workflows for LLM and multimodal training. You will ensure data quality, drive labeling consistency across projects, and collaborate with engineers and researchers to resolve edge cases that affect model behavior.

You will own dataset-level quality strategy, execute RLHF and LLM evaluation tasks, improve guidelines, and coordinate with stakeholders on data schemas, prompts, and evaluation metrics while supporting