AI Benchmark Auditor (Python & Evaluation)

HumanitApp; Northern (KY); Hybrid; USD 96,000 - 124,000; Full time

Job summary

HumanitApp is seeking a role focused on evaluating benchmark tasks used to train and assess frontier AI models, focusing on quality and reproducibility.

You will assess tasks, patches, and harnesses while delivering rubric-based feedback to ensure rigorous evaluation standards.

This remote-friendly opportunity offers compensation ranging from $70 to $90 per hour, with workload and terms clarified during the official application process.

Responsibilities

• Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate frontier AI models.
• Review repository-level tasks, reference patches, test harnesses, and grading integrity.
• Provide rubric-based written feedback.

Skills

AI evaluation

Python

Software engineering

Job description

HumanitApp is seeking a role focused on evaluating benchmark tasks used to train and assess frontier AI models, focusing on quality and reproducibility.

You will assess tasks, patches, and harnesses while delivering rubric-based feedback to ensure rigorous evaluation standards.

This remote-friendly opportunity offers compensation ranging from $70 to $90 per hour, with workload and terms clarified during the official application process.