SWE-Bench Task Auditor

Marketframe

United States

On-site

USD 140,000 - 190,000

Full time

12 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Marketframe is seeking a software engineer to evaluate the quality, correctness, and reproducibility of benchmark tasks used for frontier AI model training and evaluation. You will assess repository-level tasks, reference patches, test harnesses, and grading integrity, providing rubric-based written feedback.

The role requires 3+ years of professional software engineering experience, strong open-source involvement, and proficiency in Python plus at least one of Java, Go, TypeScript, or C++.

Qualifications

  • 3+ years professional software engineering experience.
  • Real open-source contribution or maintainer experience (merged PRs, committer/maintainer roles).
  • Strong ability to audit reference patches, test runners, and Docker isolation, and detect answer leakage / reward hacking.
  • Fluency across Python and at least one of Java / Go / TypeScript / C++.

Skills

Software engineering
Open-source contribution
Audit reference patches
Docker isolation
Python
Java/Go/TS/C++

Tools

Docker

Job description

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications
  • 3+ years professional software engineering
  • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
  • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
  • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications
  • Familiarity with SWE-Bench (Verified) or similar repository benchmarks
  • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
  • Prior code-review or task-grading experience
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

SWE-Bench Task Auditor
SWE-Bench Task Auditor

Parailabs • United States

Remote
USD 120,000 - 180,000
SWE Benchmark Auditor - Quality, Repro & Rubrics
SWE Benchmark Auditor - Quality, Repro & Rubrics

Marketframe • United States

On-site
USD 140,000 - 190,000
AI Benchmark Task Auditor & Quality Reviewer
AI Benchmark Task Auditor & Quality Reviewer

Parailabs • United States

Remote
USD 120,000 - 180,000
SWE-Bench Task Auditor (Docker, Python, Java)
SWE-Bench Task Auditor (Docker, Python, Java)

Mercor • United States

Remote
USD 96,000 - 124,000
SWE-Bench AI Task Auditor - Freelance AI Trainer Project
SWE-Bench AI Task Auditor - Freelance AI Trainer Project

Agency • United States

Remote
USD 106,000 - 144,000
SWE-Bench AI Task Auditor
SWE-Bench AI Task Auditor

Meridial • United States

Remote
USD 68,000 - 97,000
Technical Task Auditor — SWE-Bench Task Auditor
Technical Task Auditor — SWE-Bench Task Auditor

Meridial • United States

Remote
USD 83,000 - 165,000
Remote Bench Task Auditor — Docker, Python, Java
Remote Bench Task Auditor — Docker, Python, Java

Mercor • United States

Remote
USD 96,000 - 124,000
Remote SWE-Bench Task Auditor — AI Training Quality
Remote SWE-Bench Task Auditor — AI Training Quality

Agency • United States

Remote
USD 106,000 - 144,000
Remote SWE-Bench Task Auditor (Contract)
Remote SWE-Bench Task Auditor (Contract)

Meridial • United States

Remote
USD 68,000 - 97,000