Remote AI/ML Data Annotation Engineer

Rex.zone; United States; On-site; USD 41,328 - 68,880; Full time

Job summary

Rex.zone is looking for a remote, full-time role in the United States focused on developing high-quality annotation workflows for AI/ML. Responsibilities include owning training data quality metrics, designing labeling schemas, and implementing QA evaluation programs. The candidate should have over 3 years of experience in data operations or related fields and a robust understanding of NLP, LLM evaluation, and human feedback signals. The compensation ranges from $30 to $50 per hour.

Qualifications

• 3+ years in data operations, ML ops, evaluation, or annotation engineering.
• Strong understanding of NLP, LLM evaluation, and human feedback signals.
• Experience with QA evaluation methods and workflow automation.

Responsibilities

• Own training data quality metrics and improvement plans.
• Design labeling schemas and write annotation guidelines.
• Implement QA evaluation programs and perform RLHF preference labeling.

Skills

Data operations

ML ops

NLP

Evaluation methods

Technical writing

Job description

About Rex.zone

Rex.zone is a recruiting and talent solutions platform supporting modern AI/ML training workflows across the United States. This remote, full-time role focuses on building scalable annotation and evaluation programs that improve training data quality and model performance in production LLM pipelines.

Role Overview

Build and maintain high-quality annotation workflows for AI/ML, including RLHF-style preference data, prompt evaluation, and QA evaluation loops. You will design labeling schemas and write clear annotation guidelines to ensure consistent, measurable outcomes.

Responsibilities

• Own training data quality metrics, error taxonomies, and continuous improvement plans
• Design labeling schemas and write/update annotation guidelines and ambiguity-resolution rules
• Implement QA evaluation programs (review queues, spot checks, gold tasks, inter-annotator agreement)
• Perform RLHF preference labeling and ranking to support alignment and helpfulness
• Run prompt evaluation and regression test sets to track model performance improvement
• Support NLP tasks such as named entity recognition and intent classification
• Support computer vision annotation (e.g., bounding boxes, segmentation) when needed
• Execute content safety labeling for policy-aligned model behavior
• Partner with engineering to integrate tools and datasets into LLM training pipelines

Qualifications

• 3+ years in data operations, ML ops, evaluation, or annotation engineering
• Strong understanding of NLP, LLM evaluation, and human feedback signals (RLHF)
• Experience with QA evaluation methods, sampling strategies, and disagreement analysis
• Familiarity with annotation tools and workflow automation
• Excellent technical writing for guidelines and repeatable processes

Compensation

Competitive hourly base pay: $30–$50/hr.