Remote ERP AI Evaluation Specialist

AuraOne, Inc.; United States; Remote; USD 28,000 - 48,000; Part time

Job summary

AuraOne, Inc. is seeking a remote ERP Workflow AI Reviewer to assess model outputs, apply a quality rubric, and provide structured feedback for retraining. The role emphasizes edge-case labeling and rationale writing to support the modeling team.

The contractor will compare responses, tag issues, calibrate against gold-standard examples, and document recurring failures to improve future training cycles. Requires reliable weekly availability and strong attention to detail.

Qualifications

• Experience evaluating AI outputs against rubrics.
• Clear written reasoning naming issues and rubric clauses.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate ERP workflow AI evaluation model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples weekly.
• Document recurring failure modes for targeted training.

Skills

Model evaluation

Rubric annotation

Severity tagging

Inter-rater calibration

ERP workflows

Workflow evaluation

Attention to detail

Operational judgment

Team feedback

Tools

Inter-rater metrics

Job description

AuraOne, Inc. is seeking a remote ERP Workflow AI Reviewer to assess model outputs, apply a quality rubric, and provide structured feedback for retraining. The role emphasizes edge-case labeling and rationale writing to support the modeling team.

The contractor will compare responses, tag issues, calibrate against gold-standard examples, and document recurring failures to improve future training cycles. Requires reliable weekly availability and strong attention to detail.