Remote Senior AI Data Labeler - RLHF, NER & CV Annotation

Rex.zone; Austin (TX); Remote; USD 65,000 - 95,000; Full time

Job summary

Rex.zone is seeking a Senior AI data annotation professional to support LLM training pipelines from Austin, TX. This fully remote, full-time role focuses on RLHF workflows, prompt evaluation, QA checks, and content safety labeling across NLP and computer vision datasets.

You will collaborate with AI/ML stakeholders to ensure data quality, create guideline logs, and drive improvements in labeling accuracy and auditability.

Qualifications

• Mid-senior experience with annotation, labeling, or evaluation workflows.
• Strong judgment and consistency on edge cases and nuanced content.
• Experience with QA evaluation, error analysis, and training data quality improvement.
• Comfort with LLM outputs, prompt evaluation, and RLHF ranking/scoring frameworks.
• Clear written communication and documentation for auditability.

Responsibilities

• Perform expert AI data annotation and labeling for NLP, LLM training, and computer vision datasets.
• Run RLHF workflows: preference ranking, pairwise comparisons, critique writing, and rubric-based scoring.
• Execute prompt/response evaluation to identify failure modes, hallucinations, and unsafe outputs.
• Apply content safety labeling (toxicity, self-harm, hate, harassment, sexual content, policy violations).
• Conduct QA evaluation and audits to ensure annotation guideline compliance and consistent labeling.
• Create decision logs for ambiguous cases and propose guideline refinements.
• Monitor training data quality metrics and sampling strategies to improve model performance.
• Collaborate with engineering and AI/ML stakeholders to integrate labeled data into LLM training pipelines.

Skills

Annotation workflows

Edge-case judgment

QA evaluation

RLHF ranking

Clear written comms

Job description

Rex.zone is seeking a Senior AI data annotation professional to support LLM training pipelines from Austin, TX. This fully remote, full-time role focuses on RLHF workflows, prompt evaluation, QA checks, and content safety labeling across NLP and computer vision datasets.

You will collaborate with AI/ML stakeholders to ensure data quality, create guideline logs, and drive improvements in labeling accuracy and auditability.