Senior Data Labeling Lead — Remote AI/ML Quality

Rex.zone; United States; Remote; USD 41,328 - 68,880; Full time

Benefits offered by this job

Remote work

Full-time position

Job summary

Rex.zone is seeking a Senior Data Labeling professional to lead end-to-end labeling operations for AI/ML training workflows, focusing on NLP and computer vision data. You will translate product and research goals into clear annotation guidelines, run calibration sessions, and manage QA evaluation to reduce label noise and improve downstream model metrics.

This remote, full-time role requires ownership of labeling quality, collaboration with engineering, and the ability to triage edge cases to

Qualifications

• Experience leading data labeling programs with ownership of training data quality and QA evaluation.
• Familiarity with RLHF evaluation, prompt evaluation, and LLM evaluation concepts.
• Hands-on experience with named entity recognition and computer vision annotation, including guideline development and reviewer calibration.

Responsibilities

• Own annotation guidelines compliance, including taxonomy design, labeling instructions, and rubric creation for NLP and CV tasks.
• Lead RLHF evaluation and prompt evaluation queues, including preference ranking, response grading, and safety checks for LLM training pipelines.
• Run QA evaluation programs: gold set creation, blind review, inter-annotator agreement tracking, and root-cause analysis.
• Partner with engineering to operationalize labeling tooling, audit trails, and dataset versioning for repeatable production workflows.
• Triage edge cases and update guidelines to improve consistency and throughput.
• Report weekly on training data quality, defect rates, and model performance improvement signals linked to labeling changes.

Skills

Labeling program leadership

RLHF evaluation

NLP/CV annotation experience

Quality metrics design

Policy-driven labeling communication

Job description

Rex.zone is seeking a Senior Data Labeling professional to lead end-to-end labeling operations for AI/ML training workflows, focusing on NLP and computer vision data. You will translate product and research goals into clear annotation guidelines, run calibration sessions, and manage QA evaluation to reduce label noise and improve downstream model metrics.

This remote, full-time role requires ownership of labeling quality, collaboration with engineering, and the ability to triage edge cases to