Remote AI Trace Evaluation Auditor

AuraOne; United States; On-site; USD 55,104 - 82,656; Full time

Job summary

AuraOne is seeking a Debugging Trace Evaluation Specialist to operate remotely from the United States as an independent contractor. You will review AI outputs, grade workflow and tone, and flag policy gaps to guide the modeling team’s training.

The role requires reliability and strong attention to detail. Responsibilities include adjudicating disputes, maintaining calibration scores, and ensuring compliance with firm playbooks while collaborating with cross-functional teams in a remote,

Qualifications

• Direct working experience in debugging trace evaluation specialist operations on real teams for Debugging Trace Evaluation Specialist work.
• Clear written reasoning that names the policy or workflow being applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Review AI outputs against current debugging trace evaluation specialist operations workflows, playbooks, and firm policy for Debugging Trace Evaluation Specialist assignments.
• Grade tone, escalation logic, and stakeholder fit on a structured rubric.
• Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
• Capture the right next step so the modeling team can train on it.
• Adjudicate disputed treatments against published playbooks or firm guidance.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Skills

Operational review

Policy adherence

Workflow judgment

Stakeholder communication

Debugging Trace Evaluation specialist

Code review

Software testing

Agent evaluation

Debugging

Trace

Tools

AI tooling

Job description

AuraOne is seeking a Debugging Trace Evaluation Specialist to operate remotely from the United States as an independent contractor. You will review AI outputs, grade workflow and tone, and flag policy gaps to guide the modeling team’s training.

The role requires reliability and strong attention to detail. Responsibilities include adjudicating disputes, maintaining calibration scores, and ensuring compliance with firm playbooks while collaborating with cross-functional teams in a remote,