SWE-Bench Task Auditor

Parailabs

United States

Remote

USD 120,000 - 180,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Parailabs is seeking a software engineering specialist to evaluate the quality and reproducibility of benchmark tasks used to train and evaluate frontier AI models. The role involves auditing reference patches, test runners, and Docker isolation, and delivering rubric-based feedback on repository-level tasks and grading integrity.

The ideal candidate has strong Python skills, open-source contributions, and experience auditing test harnesses and reward systems in AI benchmarks.

Qualifications

  • 3+ years of professional software engineering experience.
  • Proven open-source contribution or maintainer experience.

Responsibilities

  • Evaluate quality, correctness, and reproducibility of benchmark tasks.
  • Provide rubric-based written feedback on repository tasks, patches, and test harnesses.

Skills

3+ years software engineering
Open-source contribution
Audit reference patches
Docker isolation
Python

Tools

Docker
Git

Job description

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications
  • 3+ years professional software engineering
  • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
  • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
  • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications
  • Familiarity with SWE-Bench (Verified) or similar repository benchmarks
  • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
  • Prior code-review or task-grading experience
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

SWE-Bench Task Auditor
SWE-Bench Task Auditor

Marketframe • United States

On-site
USD 140,000 - 190,000
SWE Benchmark Auditor - Quality, Repro & Rubrics
SWE Benchmark Auditor - Quality, Repro & Rubrics

Marketframe • United States

On-site
USD 140,000 - 190,000
AI Benchmark Task Auditor & Quality Reviewer
AI Benchmark Task Auditor & Quality Reviewer

Parailabs • United States

Remote
USD 120,000 - 180,000
SWE-Bench Task Auditor (Docker, Python, Java)
SWE-Bench Task Auditor (Docker, Python, Java)

Mercor • United States

Remote
USD 96,000 - 124,000
SWE-Bench AI Task Auditor - Freelance AI Trainer Project
SWE-Bench AI Task Auditor - Freelance AI Trainer Project

Agency • United States

Remote
USD 106,000 - 144,000
SWE-Bench AI Task Auditor
SWE-Bench AI Task Auditor

Meridial • United States

Remote
USD 68,000 - 97,000
Technical Task Auditor — SWE-Bench Task Auditor
Technical Task Auditor — SWE-Bench Task Auditor

Meridial • United States

Remote
USD 83,000 - 165,000
Remote Bench Task Auditor — Docker, Python, Java
Remote Bench Task Auditor — Docker, Python, Java

Mercor • United States

Remote
USD 96,000 - 124,000
Remote SWE-Bench Task Auditor — AI Training Quality
Remote SWE-Bench Task Auditor — AI Training Quality

Agency • United States

Remote
USD 106,000 - 144,000
Remote SWE-Bench Task Auditor (Contract)
Remote SWE-Bench Task Auditor (Contract)

Meridial • United States

Remote
USD 68,000 - 97,000