AI Developer Trace Task Auditor

Obsidian; San Francisco (CA); On-site; USD 120,000 - 170,000; Full time

Job summary

Obsidian seeks an experienced evaluator to assess AI-assisted software-development traces used to train frontier AI models. You will judge correctness, workflow soundness, and reasoning across end-to-end coding sessions produced with AI-assisted developer tools.

Responsibilities include rubric-based feedback and assessing trajectories for best practices, with a focus on maintaining high-quality developer tooling and workflows.

Qualifications

• 3+ years of professional software development.
• Hands-on experience with AI-assisted coding tools and agentic workflows.
• Strong reading and debugging skills across full-stack or backend systems.
• Ability to evaluate multi-step coding trajectories for correctness and best practices.

Responsibilities

• Evaluate the quality and correctness of AI-assisted development traces used for model training and evaluation.
• Provide rubric-based written feedback on coding sessions and workflows.
• Assess end-to-end coding sessions for correctness, reasoning, and best practices.

Skills

Software development

AI-assisted coding tools

Code reading & debugging

Code trajectory evaluation

Tools

Cursor

GitHub Copilot

Claude Code

Kiro

Amazon CodeCatalyst

Job description

Evaluate the quality and correctness of AI-assisted software-development traces used to train and evaluate a frontier AI lab's models. You'll assess end-to-end coding sessions produced with AI-assisted developer tools — judging correctness, workflow soundness, and reasoning — and provide clear, rubric-based written feedback.

Basic Qualifications

• 3+ years professional software development


• Hands-on experience with AI-assisted coding tools and agentic / spec-driven workflows (Cursor, GitHub Copilot, Claude Code, or similar)


• Strong code-reading and debugging skills across full-stack or backend systems


• Ability to evaluate multi-step coding trajectories for correctness and best practice

Preferred Qualifications

• Experience with Kiro or Amazon CodeCatalyst


• Prior work evaluating or grading AI-generated code


• Contributions to developer tooling