Remote AI Benchmark & Task Design Reviewer

AuraOne; United States; On-site; USD 41,000 - 76,000; Full time

Job summary

AuraOne seeks an AI Benchmark and Task Designer to work remotely as a review specialist. You will assess AI outputs against benchmarks, grade craft, and document corrections for training the modeling team. This is a contractor role requiring portfolio experience and reliable async availability for about 10 hours per week.

Responsibilities include scoring outputs, flagging tooling issues, and adjudicating judgments per firm style guides. Prior experience in AI-assisted creative tooling is a plus.

Qualifications

• Portfolio or production experience in ai benchmark and task designer creative review or an adjacent creative discipline for AI Benchmark and Task Designer work.
• Hands-on familiarity with the standard tooling for the discipline.
• Clear written reasoning that names the craft issue and the standard being applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Review AI outputs against current ai benchmark and task designer creative review craft and tooling standards for AI Benchmark and Task Designer assignments.
• Grade composition, constraint adherence, and production-readiness on a structured rubric.
• Flag tooling-specific failures (file format, scale, naming, layer hygiene) with severity tags.
• Capture the corrected approach or reference so the modeling team can train on it.
• Adjudicate disputed creative judgments against published references or firm style guides.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Skills

Creative review

Production-grade craft

Tooling fluency

Asset evaluation

AI Benchmark and Task Designer

Cross-cutting evaluation roles

AI evaluation

Rubric writing

Expert review

Job description

AuraOne seeks an AI Benchmark and Task Designer to work remotely as a review specialist. You will assess AI outputs against benchmarks, grade craft, and document corrections for training the modeling team. This is a contractor role requiring portfolio experience and reliable async availability for about 10 hours per week.

Responsibilities include scoring outputs, flagging tooling issues, and adjudicating judgments per firm style guides. Prior experience in AI-assisted creative tooling is a plus.