Senior Data Annotation Lead - Remote AI/ML Evaluation

Rex.zone; Miami (FL); On-site; USD 41,328 - 68,880; Full time

Job summary

Rex.zone is hiring for a Senior Data Annotation role in a remote-first setup targeting the Miami market. You will lead annotation guidelines, evaluate labeling quality, and perform RLHF tasks to improve AI/ML training data.

Expect collaboration with engineering to build scalable labeling pipelines and maintain consistency through documented decisions. The position emphasizes strong written communication, domain flexibility across NLP/NER, LLM evaluation, CV labeling, or content safety labeling,

Qualifications

• Mid-Senior level experience in data annotation, data labeling, or AI/ML evaluation programs.
• Strong understanding of training data quality and how labels affect model behavior.
• Proven QA evaluation experience (sampling, calibration, agreement measurement, error analysis).
• Experience in at least one domain: NLP/NER, LLM evaluation (prompt evaluation/RLHF), CV annotation, or content safety labeling.
• Clear written communication and comfort working in remote, metrics-driven environments.

Responsibilities

• Own annotation guidelines compliance and refine rubrics for high-agreement labeling.
• Perform RLHF tasks including preference ranking, justification writing, and edge-case adjudication.
• Execute prompt evaluation and response grading across safety, factuality, instruction-following, and style.
• Run QA evaluation workflows: sampling plans, error taxonomies, inter-annotator agreement checks, and corrective actions.
• Support named entity recognition and information extraction labeling for NLP datasets.
• Deliver computer vision annotation as needed (bounding boxes, polygons, keypoints, segmentation masks).
• Perform content safety labeling (toxicity, harassment, self-harm, sexual content, violence).
• Partner with engineering to improve tooling, audits, and scalable labeling pipelines.
• Document decisions and elevate ambiguous cases to maintain consistency.

Skills

Data annotation

QA evaluation

Remote work

NLP/NER

Communication

Job description

Rex.zone is hiring for a Senior Data Annotation role in a remote-first setup targeting the Miami market. You will lead annotation guidelines, evaluate labeling quality, and perform RLHF tasks to improve AI/ML training data.

Expect collaboration with engineering to build scalable labeling pipelines and maintain consistency through documented decisions. The position emphasizes strong written communication, domain flexibility across NLP/NER, LLM evaluation, CV labeling, or content safety labeling,