Senior Data Labeling Specialist - Remote, RLHF & LLM Eval

Rex.zone; San Diego (CA); On-site; USD 41,328 - 68,880; Full time

Job summary

Rex.zone in San Diego (remote) is seeking a Mid-Senior Data Labeling Specialist to elevate training data quality across NLP, computer vision, and content safety workflows. This full-time, remote role focuses on complex edge cases and high-signal labeling outcomes.

You will work independently, refine guidelines, and contribute to RLHF evaluations and adjudication notes while upholding privacy and security standards.

Qualifications

• Mid-Senior experience in data labeling, data annotation, or ML data operations.
• Follow and refine annotation guidelines with high precision.
• Experience with QA evaluation, audit sampling, and quality scoring.
• Familiarity with RLHF, preference ranking, or prompt evaluation for LLM pipelines.
• NLP tasks such as named entity recognition and classification.
• Strong written reasoning for label justifications and adjudication notes.
• Ability to work independently in a remote, full-time environment.

Responsibilities

• Execute senior-level data labeling for NLP, computer vision, and content safety datasets
• Perform RLHF preference ranking and large language model evaluation
• Run QA evaluation audits for annotation guidelines compliance and consistency
• Handle complex edge cases (ambiguity, long-context, policy-sensitive content)
• Create clear written rationales to support adjudication and error analysis
• Calibrate with teammates using gold sets, adjudication, and disagreement analysis
• Suggest tooling/workflow improvements to increase accuracy and throughput
• Maintain privacy, security, and content-handling best practices while remote

Skills

Data labeling

Quality assurance

RLHF

NLP

Remote work

Adjudication notes

Guideline refinement

Edge cases handling

Job description

Rex.zone in San Diego (remote) is seeking a Mid-Senior Data Labeling Specialist to elevate training data quality across NLP, computer vision, and content safety workflows. This full-time, remote role focuses on complex edge cases and high-signal labeling outcomes.

You will work independently, refine guidelines, and contribute to RLHF evaluations and adjudication notes while upholding privacy and security standards.