Senior Data Labeling Lead - Remote RLHF & QA

Rex.zone; Nashville (TN); Remote; USD 41,328 - 68,880; Full time

Job summary

Rex.zone is seeking a Senior Data Labeling Specialist to own end-to-end labeling workflows for AI/ML datasets across NLP, computer vision, and content safety. You will lead annotation and reviewer-level QA, and conduct RLHF-style evaluations connecting label quality to model performance.

Responsibilities include NLP and CV labeling, guideline development, and cross-functional collaboration to improve data quality and model behavior, with remote work options for US-based candidates including

Qualifications

• 3+ years in data labeling/annotation or AI/ML evaluation, including QA evaluation and reviewer workflows.
• Hands-on experience with LLM evaluation, RLHF evaluation, or structured prompt evaluation using rubrics and preference judgments.
• Strong guideline compliance, error analysis, and understanding of data quality impact on model performance.
• Experience with NLP tasks (NER, text classification) and common dataset formats/tooling.
• Familiarity with computer vision annotation concepts and ambiguity resolution patterns.
• Ability to write clear guidelines, make consistent decisions, and communicate edge cases cross-functionally.

Responsibilities

• Label and review complex text, image, and multimodal datasets while maintaining high inter-annotator agreement.
• Run QA evaluation programs: sampling plans, error taxonomy, root-cause analysis, and corrective actions tied to guideline compliance.
• Perform RLHF evaluation workflows such as pairwise preference ranking, rubric-based scoring, and prompt evaluation for instruction-following and safety behavior.
• Execute NLP labeling tasks including named entity recognition, classification, summarization evaluation, and groundedness/faithfulness checks.
• Execute computer vision annotation such as bounding boxes, polygons, keypoints, segmentation, and attribute tagging; validate edge cases.
• Support content safety labeling (e.g., harassment, self-harm, hate) with consistent escalation paths and audit trails.
• Partner with engineering/ML teams to translate model failure modes into updated guidelines, new label schemas, and targeted re-annotation.
• Document decisions, dataset specs, and QA findings to support reproducibility and dataset lineage.

Skills

Data labeling

LLM evaluation

RLHF evaluation

NLP tasks

CV annotation

Guideline development

Job description

Rex.zone is seeking a Senior Data Labeling Specialist to own end-to-end labeling workflows for AI/ML datasets across NLP, computer vision, and content safety. You will lead annotation and reviewer-level QA, and conduct RLHF-style evaluations connecting label quality to model performance.

Responsibilities include NLP and CV labeling, guideline development, and cross-functional collaboration to improve data quality and model behavior, with remote work options for US-based candidates including