SWE-Bench Task Auditor (Docker, Python, Java)

Mercor

United States

Remote

USD 96,000 - 124,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

DigiNo is seeking a software engineering reviewer to assess benchmark tasks used to test AI models. You will verify that code fixes, tests, and grading setup are correct and reproducible across environments.

The ideal candidate has at least three years of professional engineering experience, a history of merged open-source contributions, and strong Python skills plus one other major language. This 100% remote role is US-only and full-time at 40 hours per week.

Qualifications

  • Requires at least three years of professional software engineering experience.
  • Merged open-source contributions encouraged.
  • Strong Python plus one other major language.

Responsibilities

  • Review benchmark tasks used to test AI models and verify code fixes, tests, and grading setup for correctness and reproducibility.

Job description

You will review software engineering benchmark tasks used to test AI models, checking that the code fixes, tests, and grading setup are correct and can be reproduced. Strong fits have at least three years of professional engineering, merged open-source contributions, and solid Python plus one other major language. The role is remote for candidates in the United States, at 40 hours a week. Pay is listed at $70 to $90/hr as of 2026-10-14.

This is a 100% remote role. Applicants must live in the United States.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote Bench Task Auditor — Docker, Python, Java
Remote Bench Task Auditor — Docker, Python, Java

Mercor • United States

Remote
USD 96,000 - 124,000
SWE-Bench AI Task Auditor
SWE-Bench AI Task Auditor

Meridial • United States

Remote
USD 68,000 - 97,000
Remote SWE-Bench Task Auditor — AI Training Quality
Remote SWE-Bench Task Auditor — AI Training Quality

Agency • United States

Remote
USD 106,000 - 144,000
Remote SWE-Bench Task Auditor (Contract)
Remote SWE-Bench Task Auditor (Contract)

Meridial • United States

Remote
USD 68,000 - 97,000
SWE-Bench AI Task Auditor - Freelance AI Trainer Project
SWE-Bench AI Task Auditor - Freelance AI Trainer Project

Agency • United States

Remote
USD 106,000 - 144,000
SWE-Bench Task Auditor
SWE-Bench Task Auditor

Marketframe • United States

On-site
USD 140,000 - 190,000
SWE-Bench Task Auditor
SWE-Bench Task Auditor

Parailabs • United States

Remote
USD 120,000 - 180,000
SWE Benchmark Auditor - Quality, Repro & Rubrics
SWE Benchmark Auditor - Quality, Repro & Rubrics

Marketframe • United States

On-site
USD 140,000 - 190,000
AI Coding Benchmark Engineer — Remote, $1k per Task
AI Coding Benchmark Engineer — Remote, $1k per Task

NoDesk • United States

Remote
USD 39,000 - 52,000
AI Benchmark Task Auditor & Quality Reviewer
AI Benchmark Task Auditor & Quality Reviewer

Parailabs • United States

Remote
USD 120,000 - 180,000