Remote Supply Chain Evaluation Specialist

AuraOne; United States; On-site; USD 55,104 - 82,656; Part time

Job summary

AuraOne is seeking a remote Contractor to evaluate supply chain planning agent prompts and responses. You will compare outputs to the AuraOne rubric, assign severity tags, and provide structured feedback for retraining.

The role emphasizes labeling edge cases, maintaining calibration against gold standards, and documenting recurring failure modes to guide model improvements. Applicants must be authorized to work from the US and be available for at least 10 hours per week.

Qualifications

• Experience evaluating model outputs against rubrics.
• Ability to name issues and apply rubric clauses.
• Attention to detail and ability to flag problems in prompts.

Responsibilities

• Evaluate model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and provide a rationale for the stronger answer.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Capture ambiguous prompts and route for rubric updates.
• Calibrate reviewer scores against gold-standard examples weekly.
• Document recurring failure modes for model improvement.

Skills

Model evaluation

Rubric annotation

Severity tagging

Inter-rater calibration

Supply Chain Planning Agent evaluation

Workflow evaluation

Operational judgment

Job description

AuraOne is seeking a remote Contractor to evaluate supply chain planning agent prompts and responses. You will compare outputs to the AuraOne rubric, assign severity tags, and provide structured feedback for retraining.

The role emphasizes labeling edge cases, maintaining calibration against gold standards, and documenting recurring failure modes to guide model improvements. Applicants must be authorized to work from the US and be available for at least 10 hours per week.