SWE-Bench AI Task Auditor - Freelance AI Trainer Project

Agency; United States; Remote; USD 146,026,000 - 198,374,000; Full time

Job summary

Agency is seeking experienced software engineering specialists to audit tasks used for training and evaluating AI systems. You will assess technical accuracy, realism, and solvability, and provide actionable feedback on codebase integration issues and test failures.

Role is a freelance contract with remote work. Pay is $60 per hour, with rate determined after evaluating experience and location.

Qualifications

• Proven experience in software engineering and navigating complex codebases.
• Deep knowledge of SWE-Bench standards and evaluation criteria.
• Strong analytical and problem-solving skills to test and troubleshoot complex technical scenarios.

Responsibilities

• Audit tasks used to train and evaluate AI systems for technical accuracy and realism.
• Provide clear, actionable feedback on codebase integration issues, test failures, or logic errors.

Skills

Software engineering

SWE-Bench standards

Analytical skills

Job description

Project Overview

We are sourcing experienced technical specialists in Software Engineering (SWE-Bench) to audit tasks used to train and evaluate AI systems. The objective of this project is to ensure that AI training workflows and tasks are technically rigorous, practical, and highly accurate within your specific area of expertise.

Project Deliverables & Scope

Task Evaluation: Assess whether software engineering tasks are technically accurate, realistic, solvable, reproducible, and supported by reliable tests and evaluation criteria. Technical Auditing & Feedback: Provide clear, actionable feedback on any identified codebase integration issues, test failures, or logic errors within the tasks.

Required Expertise

Demonstrable professional experience and deep technical knowledge in software engineering (including complex codebase navigation, real-world application development, and SWE-Bench standards). (Note: Candidates will be considered for one specialty based on their experience; expertise across other domains is not required). Strong analytical and problem-solving skills to rigorously test, troubleshoot, and evaluate complex technical scenarios.

Pay & Contract

We offer a pay rate of $60 per hour, with the exact rate determined after evaluating your experience, expertise, and geographic location. As a contractor you'll supply a secure computer and high-speed internet; company-sponsored benefits such as health insurance and PTO do not apply. Employment type: Freelance Contract Workplace type: Remote