VS Code macOS Evaluation Specialist (AI Data Reviewer)

AuraOne, Inc.; United States; Remote; USD 76,000 - 90,000; Full time

Job summary

AuraOne, Inc. seeks a remote contractor to review application users' Visual Studio Code on macOS outputs. You will assess model responses against a versioned rubric, assign severity tags, and provide written rationales for each decision.

You will compare paired results, label failures, and document common issues to help retrain the system. This role requires precise reasoning and consistent rubric application.

Qualifications

• Prior evaluation, annotation, or human-rater experience on code evaluation content.
• Ability to apply multi-page rubrics consistently.
• Clear written reasoning that names the issue and rubric clause.
• Strong attention to detail.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate model outputs against the rubric and assign severity tags.
• Compare paired responses and provide rationale.
• Label hallucinations, instruction-following failures, and unsafe content.
• Document recurring failure modes for rubric updates.
• Calibrate scores against gold-standard examples weekly.
• Audit batches for rubric consistency and report drift.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

MacOS code evaluation

Job description

AuraOne, Inc. seeks a remote contractor to review application users' Visual Studio Code on macOS outputs. You will assess model responses against a versioned rubric, assign severity tags, and provide written rationales for each decision.

You will compare paired results, label failures, and document common issues to help retrain the system. This role requires precise reasoning and consistent rubric application.