AI Code Trace Auditor for Developer Tools

Obsidian; San Francisco (CA); On-site; USD 120,000 - 170,000; Full time

Job summary

Obsidian seeks an experienced evaluator to assess AI-assisted software-development traces used to train frontier AI models. You will judge correctness, workflow soundness, and reasoning across end-to-end coding sessions produced with AI-assisted developer tools.

Responsibilities include rubric-based feedback and assessing trajectories for best practices, with a focus on maintaining high-quality developer tooling and workflows.

Qualifications

• 3+ years of professional software development.
• Hands-on experience with AI-assisted coding tools and agentic workflows.
• Strong reading and debugging skills across full-stack or backend systems.
• Ability to evaluate multi-step coding trajectories for correctness and best practices.

Responsibilities

• Evaluate the quality and correctness of AI-assisted development traces used for model training and evaluation.
• Provide rubric-based written feedback on coding sessions and workflows.
• Assess end-to-end coding sessions for correctness, reasoning, and best practices.

Skills

Software development

AI-assisted coding tools

Code reading & debugging

Code trajectory evaluation

Tools

Cursor

GitHub Copilot

Claude Code

Kiro

Amazon CodeCatalyst

Job description

Obsidian seeks an experienced evaluator to assess AI-assisted software-development traces used to train frontier AI models. You will judge correctness, workflow soundness, and reasoning across end-to-end coding sessions produced with AI-assisted developer tools.

Responsibilities include rubric-based feedback and assessing trajectories for best practices, with a focus on maintaining high-quality developer tooling and workflows.