AI Developer Trace Task Auditor

Mercor; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Mercor in San Francisco seeks an experienced software development evaluator to assess AI-assisted coding traces used to train and evaluate frontier AI models. You will review end-to-end coding sessions produced with AI-assisted developer tools, judging correctness, workflow soundness, and reasoning, and deliver clear rubric-based feedback.

The role emphasizes collaboration with ML researchers and software engineers; you will apply structured criteria, document findings, and help improve tooling

Qualifications

• 3+ years professional software development.
• Hands-on experience with AI-assisted coding tools and agentic workflows.
• Strong code-reading and debugging across full-stack or backend systems.
• Ability to evaluate multi-step coding trajectories for correctness and best practice.

Responsibilities

• Evaluate the quality and correctness of AI-assisted software-development traces used to train and evaluate models.
• Judging correctness, workflow soundness, and reasoning of end-to-end coding sessions.
• Provide rubric-based written feedback.

Skills

AI-assisted coding tools

Code-reading & debugging

Evaluating multi-step trajectories

Full-stack or backend proficiency

Tools

Cursor

GitHub Copilot

Claude Code

Amazon CodeCatalyst

Kiro

Job description

Evaluate the quality and correctness of AI-assisted software-development traces used to train and evaluate a frontier AI lab's models. You'll assess end-to-end coding sessions produced with AI-assisted developer tools — judging correctness, workflow soundness, and reasoning — and provide clear, rubric-based written feedback.

Basic Qualifications

• 3+ years professional software development


• Hands-on experience with AI-assisted coding tools and agentic / spec-driven workflows (Cursor, GitHub Copilot, Claude Code, or similar)


• Strong code-reading and debugging skills across full-stack or backend systems


• Ability to evaluate multi-step coding trajectories for correctness and best practice

Preferred Qualifications

• Experience with Kiro or Amazon CodeCatalyst


• Prior work evaluating or grading AI-generated code


• Contributions to developer tooling