AI Benchmark Task Auditor & Quality Reviewer

Parailabs

United States

À distance

USD 120 000 - 180 000

Plein temps

14 jours+

Une candidature complète en une minute

Un CV et une lettre de motivation personnalisés, prêts à envoyer.

Passez les filtres ATS

Résumé du poste

Parailabs is seeking a software engineering specialist to evaluate the quality and reproducibility of benchmark tasks used to train and evaluate frontier AI models. The role involves auditing reference patches, test runners, and Docker isolation, and delivering rubric-based feedback on repository-level tasks and grading integrity.

The ideal candidate has strong Python skills, open-source contributions, and experience auditing test harnesses and reward systems in AI benchmarks.

Qualifications

  • 3+ years of professional software engineering experience.
  • Proven open-source contribution or maintainer experience.

Responsabilités

  • Evaluate quality, correctness, and reproducibility of benchmark tasks.
  • Provide rubric-based written feedback on repository tasks, patches, and test harnesses.

Connaissances

3+ years software engineering
Open-source contribution
Audit reference patches
Docker isolation
Python

Outils

Docker
Git

Description du poste

Parailabs is seeking a software engineering specialist to evaluate the quality and reproducibility of benchmark tasks used to train and evaluate frontier AI models. The role involves auditing reference patches, test runners, and Docker isolation, and delivering rubric-based feedback on repository-level tasks and grading integrity.

The ideal candidate has strong Python skills, open-source contributions, and experience auditing test harnesses and reward systems in AI benchmarks.

Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

SWE Benchmark Auditor - Quality, Repro & Rubrics
SWE Benchmark Auditor - Quality, Repro & Rubrics

Marketframe • États-Unis

Sur place
USD 140 000 - 190 000
SWE-Bench Task Auditor
SWE-Bench Task Auditor

Parailabs • États-Unis

À distance
USD 120 000 - 180 000
SWE-Bench Task Auditor
SWE-Bench Task Auditor

Marketframe • États-Unis

Sur place
USD 140 000 - 190 000
AI Code Quality Auditor & Feedback Architect
AI Code Quality Auditor & Feedback Architect

Parailabs • États-Unis

À distance
USD 120 000 - 190 000
Remote AI Benchmark QC & Coding Auditor
Remote AI Benchmark QC & Coding Auditor

Remolity, Inc. • États-Unis

À distance
USD 21 000 - 34 000
Paid pilot
Flexible scheduling
Remote work within eligible countries
Remote Bench Task Auditor — Docker, Python, Java
Remote Bench Task Auditor — Docker, Python, Java

Mercor • États-Unis

À distance
USD 96 000 - 124 000
AI Benchmark Engineer (Remote Contract)
AI Benchmark Engineer (Remote Contract)

LinkedIn • États-Unis

À distance
USD 83 000 - 165 000
AI Benchmark QC Auditor — Remote (North America)
AI Benchmark QC Auditor — Remote (North America)

Verita Ai • États-Unis

À distance
USD 9 100 - 23 000
AI Benchmarking & Evaluation Scientist
AI Benchmarking & Evaluation Scientist

snorkelai • États-Unis

À distance
USD 120 000 - 180 000
Remote AI Code Benchmark Engineer
Remote AI Code Benchmark Engineer

Turing • États-Unis

À distance
USD 83 000 - 152 000
Fully remote
Cutting-edge AI projects