Software Benchmark Auditor: Reproducibility & Quality

Obsidian; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Obsidian in San Francisco seeks a reviewer and assessor of benchmark tasks for frontier AI labs. You will evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate models, and review repository-level tasks, reference patches, and test harnesses.

You will provide rubric-based written feedback, detect leakage or reward hacking, and help improve grading integrity across the benchmark suite.

Qualifications

• 3+ years of professional software engineering experience.
• Real open-source contribution or maintainer experience with merged PRs.
• Strong ability to audit reference patches, test runners, and Docker isolation.
• Fluency in Python and at least one of Java, Go, TypeScript, or C++.

Skills

Software engineering

Open-source contribution

Code auditing

Python

Other languages (Java/Go/TypeScript/C\

Tools

Docker

Test harness tooling

Job description

Obsidian in San Francisco seeks a reviewer and assessor of benchmark tasks for frontier AI labs. You will evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate models, and review repository-level tasks, reference patches, and test harnesses.

You will provide rubric-based written feedback, detect leakage or reward hacking, and help improve grading integrity across the benchmark suite.