AI Benchmark Quality Engineer for Task Evaluation

Obsidian; San Francisco (CA); Remote; USD 115,000 - 160,000; Full time

Job summary

Obsidian is seeking a candid evaluator to assess the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train frontier AI labs. You will review repository-level tasks, reference patches, test harnesses, and grading integrity, and provide rubric-based written feedback.

Responsibilities include identifying gaps in tooling, ensuring task reproducibility across environments, and delivering concrete, rubric-based assessments to guide model training and

Qualifications

• 3+ years of professional software engineering experience.
• Proven open-source contributions or maintainer roles.
• Ability to audit reference patches, test runners, and Docker isolation.
• Fluency in Python and at least one of Java/Go/TypeScript/C++.

Skills

Software engineering

Open-source maintainer

Patch audit

Python & another language

Tools

Docker

Job description

Obsidian is seeking a candid evaluator to assess the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train frontier AI labs. You will review repository-level tasks, reference patches, test harnesses, and grading integrity, and provide rubric-based written feedback.

Responsibilities include identifying gaps in tooling, ensuring task reproducibility across environments, and delivering concrete, rubric-based assessments to guide model training and