AI Developer Trace Task Auditor Mercor · Remote — United States $70-90/hr →

Dorado; Northern (KY); Hybrid; USD 90,000 - 130,000; Full time

Job summary

Dorado is seeking an experienced software-evaluation specialist to assess AI-assisted development traces used for training frontier AI models. You will judge correctness, workflow soundness, and reasoning across end-to-end coding sessions produced with AI-enabled tools.

Responsibilities include delivering rubric-based feedback, identifying gaps, and guiding tooling improvements. A strong background in software development and debugging is essential for success.

Qualifications

• 3+ years of professional software development experience.
• Hands-on experience with AI-assisted coding tools and agentic / spec-driven workflows (Cursor, GitHub Copilot, Claude Code, or similar).
• Strong code-reading and debugging skills across full-stack or backend systems.
• Ability to evaluate multi-step coding trajectories for correctness and best practices.

Responsibilities

• Evaluate AI-assisted coding traces for correctness and best practices.
• Assess end-to-end coding sessions produced with AI-assisted developer tools.
• Provide rubric-based written feedback on workflow soundness and reasoning.
• Identify gaps and provide actionable improvement guidance.

Skills

Software development

AI-assisted coding tools

Code reading

Debugging

Evaluation

Tools

Cursor

GitHub Copilot

Claude Code

Kiro

Amazon CodeCatalyst

Job description

Evaluate the quality and correctness of AI-assisted software-development traces used to train and evaluate a frontier AI lab's models. You'll assess end-to-end coding sessions produced with AI-assisted developer tools — judging correctness, workflow soundness, and reasoning — and provide clear, rubric-based written feedback.

Basic Qualifications

• 3+ years professional software development


• Hands-on experience with AI-assisted coding tools and agentic / spec-driven workflows (Cursor, GitHub Copilot, Claude Code, or similar)


• Strong code-reading and debugging skills across full-stack or backend systems


• Ability to evaluate multi-step coding trajectories for correctness and best practice

Preferred Qualifications

• Experience with Kiro or Amazon CodeCatalyst


• Prior work evaluating or grading AI-generated code


• Contributions to developer tooling