STEM Careers Canada

Rex.zone; United States; Remote; USD 41,328 - 68,880; Full time

Job summary

A tech company specializing in AI solutions is searching for a remote, full-time candidate to support AI/ML training workflows. In this role, you will perform high-accuracy data labeling, execute RLHF evaluation, and ensure training data quality. Candidates should have mid-senior experience in evaluation or data annotation, familiarity with NLP concepts, and strong communication skills. Compensation ranges from $30 to $50 per hour depending on experience.

Qualifications

• Mid-Senior experience in evaluation or annotation programs.
• Strong consistency applying detailed rubrics with low error rates.
• Familiarity with LLM training pipelines and preference-based grading.

Responsibilities

• Perform high-accuracy data labeling for text, image, and multimodal datasets.
• Execute RLHF evaluation and rubric-based scoring.
• Maintain annotation guidelines compliance.

Skills

Evaluation experience

Data annotation

Attention to detail

NLP knowledge

Job description

About The Role

This remote, full-time role supports AI/ML training workflows through high-quality data labeling, RLHF-style preference ranking, prompt evaluation, and QA evaluation. You will apply clear rubrics and annotation guidelines to improve training data quality and measurable model performance.

Key Responsibilities

• Perform high-accuracy data labeling for text, image, and multimodal datasets
• Execute RLHF evaluation (preference comparisons, rubric-based scoring)
• Run prompt evaluation and response grading for LLM evaluation
• Apply content safety labeling and policy-aware categorization
• Maintain annotation guidelines compliance and propose guideline improvements
• Conduct QA evaluation, audits, and disagreement resolution to ensure training data quality
• Document edge cases clearly and support distributed, asynchronous collaboration

Required Qualifications

• Mid-Senior experience in evaluation, annotation programs, or engineering-adjacent data operations
• Strong consistency applying detailed rubrics with low error rates
• Familiarity with LLM training pipelines and preference-based grading (RLHF)
• Working knowledge of NLP concepts (e.g., named entity recognition)
• Exposure to computer vision annotation workflows preferred
• Strong written communication for QA notes and edge-case documentation

Compensation

Competitive base pay: $30–$50 per hour.

Remote, full-time; async-first collaboration.