Frontier AI Benchmark Task Auditor

Mercor; United States; Remote; USD 110,000 - 170,000; Full time

Job summary

Mercor seeks a professional to evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate frontier AI models. You will assess repository-level tasks, reference patches, test harnesses, and grading integrity.

You will deliver clear rubric-based written feedback, outlining strengths, gaps, and concrete improvements to ensure reliable benchmarking and reproducibility across teams working with open-source tools.

Qualifications

• 3+ years of professional software engineering experience.
• Real open-source contribution or maintainer experience.
• Ability to audit reference patches, test runners, and Docker isolation.
• Fluency in Python and at least one of Java / Go / TypeScript / C++.

Responsibilities

• Evaluate benchmark task quality, correctness, and reproducibility for frontier AI models.
• Assess repository-level tasks, reference patches, test harnesses, and grading integrity.
• Provide rubric-based written feedback.

Skills

Software engineering

Open-source contribution

Code review

Docker auditing

Python

Multi-language ecosystem

Tools

Docker

Job description

Mercor seeks a professional to evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate frontier AI models. You will assess repository-level tasks, reference patches, test harnesses, and grading integrity.

You will deliver clear rubric-based written feedback, outlining strengths, gaps, and concrete improvements to ensure reliable benchmarking and reproducibility across teams working with open-source tools.