Senior Software Benchmark Auditor - Code Quality & Integrity

Mercor; San Francisco (CA); On-site; USD 150,000 - 190,000; Full time

Job summary

Mercor seeks a software engineering quality evaluator to assess benchmark tasks used to train frontier AI models. You will audit repository-level tasks, reference patches, test harnesses, and grading integrity, providing rubric-based written feedback.

The role requires 3+ years in software engineering, open-source contribution or maintainer experience, and strong ability to audit patches, test runners, and Docker isolation.

Qualifications

• 3+ years of professional software engineering.
• Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles).
• Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking.
• Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++).

Responsibilities

• Evaluate the quality, correctness, and reproducibility of benchmark tasks used to train frontier AI models.
• Provide rubric-based written feedback and assess repository-level tasks, reference patches, test harnesses, and grading integrity.

Skills

Software engineering

Open-source contribution

Code review

Audit reference patches

Test harnesses

Python

Java

Go

Tools

Docker

CI/CD

Job description

Mercor seeks a software engineering quality evaluator to assess benchmark tasks used to train frontier AI models. You will audit repository-level tasks, reference patches, test harnesses, and grading integrity, providing rubric-based written feedback.

The role requires 3+ years in software engineering, open-source contribution or maintainer experience, and strong ability to audit patches, test runners, and Docker isolation.