SWE Benchmark Auditor for AI Model Evaluation

Dorado; Northern (KY); Hybrid; USD 100,000 - 150,000; Full time

Job summary

Dorado seeks a software engineering quality assessor to evaluate the quality, correctness, and reproducibility of benchmark tasks used for frontier AI lab models. You will audit repository-level tasks, reference patches, test harnesses, and grading integrity, and provide rubric-based feedback.

Ideal candidates will have 3+ years in professional software engineering, real open-source contribution or maintainer experience, and fluency in Python plus at least one of Java, Go, TypeScript, or C++.

Qualifications

• 3+ years professional software engineering.
• Real open-source contribution or maintainer experience (merged PRs, committer/maintainer roles).
• Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking.
• Fluency across Python and at least one of Java / Go / TypeScript / C++.

Responsibilities

• Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate frontier AI lab models.
• Assess repository-level tasks, reference patches, test harnesses, and grading integrity.
• Provide rubric-based written feedback to guide improvements and ensure robust benchmarking practices.

Skills

Software engineering

Open-source contributions

Code review

Patching auditing

Python

Java

Go

TypeScript

C++

Tools

Docker

Job description

Dorado seeks a software engineering quality assessor to evaluate the quality, correctness, and reproducibility of benchmark tasks used for frontier AI lab models. You will audit repository-level tasks, reference patches, test harnesses, and grading integrity, and provide rubric-based feedback.

Ideal candidates will have 3+ years in professional software engineering, real open-source contribution or maintainer experience, and fluency in Python plus at least one of Java, Go, TypeScript, or C++.