Remote PR Reasoning Evaluator & Feedback Architect

AI Trainer Jobs; United States; Remote; USD 34,000 - 55,000; Part time

Job summary

AuraOne seeks a remote Pull Request Reasoning Workflow Reviewer to evaluate PR reasoning outputs against a defined rubric. You will help label edge cases, provide structured feedback, and support retraining efforts for the modeling team.

As a contractor, you will compare responses, justify selections, and flag issues with precise reasoning and severity tags. This role requires strong attention to detail and async availability.

Qualifications

• Experience evaluating model outputs using rubrics and tagging failures.
• Ability to annotate and label edge cases with structured feedback.
• Strong written reasoning and attention to detail.

Responsibilities

• Evaluate PR reasoning workflow outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger one with a rationale.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Capture ambiguous prompts and route for rubric updates.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Pull Request Reasoning Workflow

Code review

Software testing

Agent evaluation

Pull

Request

Reasoning

Job description

AuraOne seeks a remote Pull Request Reasoning Workflow Reviewer to evaluate PR reasoning outputs against a defined rubric. You will help label edge cases, provide structured feedback, and support retraining efforts for the modeling team.

As a contractor, you will compare responses, justify selections, and flag issues with precise reasoning and severity tags. This role requires strong attention to detail and async availability.