AI Trace Evaluator: Quality & Reasoning Auditor

HumanitApp; United States; Hybrid; USD 96,000 - 124,000; Full time

Job summary

HumanitApp is seeking an AI-evaluation professional to assess AI-assisted software-development traces used to train and evaluate frontier AI models. You will review end-to-end coding sessions produced with AI developer tools, judging correctness, workflow soundness, and reasoning.

The role offers compensation in the $70-$90 per hour range, with a 40-hour week; final rate, workload, and payment terms will be confirmed during the official application process.

Qualifications

• Experience in AI evaluation.

Responsibilities

• Evaluate AI-assisted software-development traces for accuracy, workflow soundness, and reasoning.

Job description

Future projects, not immediate vacancies. Check each network's country and experience requirements.

Evaluate the quality and correctness of AI-assisted software-development traces used to train and evaluate a frontier AI lab's models. You'll assess end-to-end coding sessions produced with AI-assisted developer tools

judging correctness, workflow soundness, and reasoning

and provide clear, rubric-based written feedback.

Basic Qualifications • 3+ years professional software development • Hands-on experience with AI-assisted coding tools and agentic / spec-driven workflows (Cursor, GitHub Copilot, Claude Code, or similar) • Strong code-reading and debugging skills across full-stack or backend systems • Ability to evaluate multi-step coding trajectories for correctness and best practice

Preferred Qualifications • Experience with Kiro or Amazon CodeCatalyst • Prior work evaluating or grading AI-generated code • Contributions to developer tooling

Mercor eligibility fields: Eligible work locations: United States.

AI evaluation

Compensation context

The listing states $70 - $90 / hour. Confirm the final rate, workload, and payment terms during the official application process.

• Ability to evaluate multi-step coding trajectories for correctness and best practice

Not the right fit?

Use your CV to find relevant roles on HumanitApp. This is separate from a partner application.

Optional role alerts from HumanitApp. Subscribe only if you want weekly emails.

Can I start immediately through an Instant Work Offer?

This listing is not a promise of immediate work. Mercor may send separate offers to prequalified candidates. Read how Instant Work Offers work .

How should I compare the listed pay?

The listing states $70 - $90 / hour. Confirm the final rate, workload, and payment terms during the official application process.

Does Apply use a referral link?

Yes. The button opens the exact verified Mercorlisting using the referral URL published for this role.

Is HumanitApp the employer?

No. HumanitApp independently curates the opportunity. The partner platform manages applications and hiring decisions.

Must I share my details here?

No. Choose the primary Apply action to continue directly without giving HumanitApp your name or email address.