AI Coding Trace Auditor - Rubric-Based Feedback

Mercor; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Mercor in San Francisco seeks an experienced software development evaluator to assess AI-assisted coding traces used to train and evaluate frontier AI models. You will review end-to-end coding sessions produced with AI-assisted developer tools, judging correctness, workflow soundness, and reasoning, and deliver clear rubric-based feedback.

The role emphasizes collaboration with ML researchers and software engineers; you will apply structured criteria, document findings, and help improve tooling

Qualifications

• 3+ years professional software development.
• Hands-on experience with AI-assisted coding tools and agentic workflows.
• Strong code-reading and debugging across full-stack or backend systems.
• Ability to evaluate multi-step coding trajectories for correctness and best practice.

Responsibilities

• Evaluate the quality and correctness of AI-assisted software-development traces used to train and evaluate models.
• Judging correctness, workflow soundness, and reasoning of end-to-end coding sessions.
• Provide rubric-based written feedback.

Skills

AI-assisted coding tools

Code-reading & debugging

Evaluating multi-step trajectories

Full-stack or backend proficiency

Tools

Cursor

GitHub Copilot

Claude Code

Amazon CodeCatalyst

Kiro

Job description

Mercor in San Francisco seeks an experienced software development evaluator to assess AI-assisted coding traces used to train and evaluate frontier AI models. You will review end-to-end coding sessions produced with AI-assisted developer tools, judging correctness, workflow soundness, and reasoning, and deliver clear rubric-based feedback.

The role emphasizes collaboration with ML researchers and software engineers; you will apply structured criteria, document findings, and help improve tooling