Remote SWE-Bench Auditor - Part-Time, $70-$90/hr

OpenTrain AI; Northern (KY); Hybrid; USD 96,000 - 124,000; Part time

Benefits offered by this job

Remote contract role (US)

40 hours/week target (20+ hours)

Competitive hourly rate $70–$90

Job summary

OpenTrain is seeking a SWE-Bench Task Auditor to evaluate repository-level software-engineering benchmark tasks used to train and assess AI models. You will examine task specifications, reference patches, test harnesses, Docker isolation, and grading integrity.

The role requires at least three years of software engineering experience, open-source contributions, and the ability to audit patches, test runners, and grading systems.

Qualifications

• At least three years of professional software-engineering experience
• Open-source contribution or maintainer experience with merged PRs or committer responsibilities
• Ability to audit reference patches, test runners, Docker isolation, and grading integrity
• Fluency in Python and at least one of Java, Go, TypeScript, or C++
• Judgment in detecting answer leakage and reward hacking in software-engineering evaluations
• Familiarity with SWE-Bench Verified or similar benchmarks is helpful
• Prior code-review or task-grading experience is helpful
• Maintainer history on major Python open-source projects is valuable

Responsibilities

• Review repository-level tasks for quality, correctness, and reproducibility
• Audit reference patches and determine whether they address the task
• Inspect test runners, grading behavior, and containerized execution for reliability
• Assess Docker isolation and overall grading integrity
• Identify answer leakage, reward hacking, and other evaluation weaknesses
• Write concise, rubric-based feedback describing findings and recommendations

Skills

Python

Java

Go

TypeScript

C++

Open-source contribution

Code review

Auditing patches

Planning and judgment

Tools

Docker

Job description

OpenTrain is seeking a SWE-Bench Task Auditor to evaluate repository-level software-engineering benchmark tasks used to train and assess AI models. You will examine task specifications, reference patches, test harnesses, Docker isolation, and grading integrity.

The role requires at least three years of software engineering experience, open-source contributions, and the ability to audit patches, test runners, and grading systems.