Senior Data Annotation & RLHF QA Specialist

Rex.zone; Boston (MA); On-site; USD 70,000 - 110,000; Full time

Job summary

Rex.zone is seeking Senior Data Annotation professionals to support AI/ML training workflows with high-quality labeled data, RLHF judgments, and QA signals for LLM evaluation and computer vision annotation. This is a remote, full-time role aligned to Boston hiring demand, focusing on training data quality and auditable labeling operations.

Join a team that emphasizes clear rationale for decisions, robust audit trails, and rigorous QA across NLP and CV tasks.

Qualifications

• Experience in data annotation, data labeling, or model evaluation.
• Ability to provide clear rationale for edge cases and maintain audit trails.
• Familiarity with data quality metrics and QA evaluation workflows.
• Strong written communication for audit trails and documentation.
• Ability to maintain throughput while preserving high precision and consistency.

Responsibilities

• Own complex annotation queues and resolve edge cases with clear rationale
• Perform RLHF preference ranking, critique labeling, and prompt evaluation
• Execute QA evaluation checks, adjudicate disagreements, and document decisions
• Complete named entity recognition and taxonomy tagging for NLP datasets
• Support computer vision annotation (e.g., bounding boxes, polygons, keypoints) as needed
• Create gold sets, run calibration sessions, and track inter-annotator agreement
• Improve annotation guidelines compliance through feedback to operations and engineering
• Surface systematic issues impacting LLM training pipelines and model performance

Skills

Data annotation

RLHF judgments

QA evaluation

NLP tagging

CV annotation

Auditable labeling

Documentation

Job description

Rex.zone is seeking Senior Data Annotation professionals to support AI/ML training workflows with high-quality labeled data, RLHF judgments, and QA signals for LLM evaluation and computer vision annotation. This is a remote, full-time role aligned to Boston hiring demand, focusing on training data quality and auditable labeling operations.

Join a team that emphasizes clear rationale for decisions, robust audit trails, and rigorous QA across NLP and CV tasks.