Remote Full-Stack Evaluator & JSON Schema Reviewer

AI Trainer Jobs; United States; Remote; USD 55,000 - 58,000; Part time

Benefits offered by this job

Remote work

Contractor engagement

US-eligible

Job summary

AI Trainer Jobs is seeking a remote evaluator for Full-Stack Engineer evaluation prompts and responses, aligning outputs with AuraOne's quality rubric. You will review paired results, label edge cases, and craft structured feedback for model retraining.

As a contractor, you will assess compliance, tag severities, and help build auditable datasets. The role emphasizes precise reasoning, attention to detail, and consistent application of rubric clauses on a weekly cadence.

Qualifications

• Prior evaluation or annotation experience on software prompts or content.
• Ability to apply rubrics and label edge cases.
• Strong written reasoning and problem description skills.

Responsibilities

• Evaluate model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the better one with rationale.
• Label hallucinations and unsafe content with structured tags.
• Capture ambiguous prompts and route for rubric updates.
• Calibrate reviewer quality against gold-standard examples weekly.

Skills

Evaluation experience

Annotation

Inter-rater calibration

Attention to detail

Clear written reasoning

Async availability

Tools

JSON Schema

JSON

API Gateway

Schema Design

Data Validation

Job description

AI Trainer Jobs is seeking a remote evaluator for Full-Stack Engineer evaluation prompts and responses, aligning outputs with AuraOne's quality rubric. You will review paired results, label edge cases, and craft structured feedback for model retraining.

As a contractor, you will assess compliance, tag severities, and help build auditable datasets. The role emphasizes precise reasoning, attention to detail, and consistent application of rubric clauses on a weekly cadence.