Software Engineers: Paid Code Review for AI Agent Evaluation

Terac; United States; Remote; USD 138,000 - 413,000; Part time

Benefits offered by this job

Remote study participation

Compensation for time

Job summary

Terac is conducting a remote, paid study to evaluate coding environments and evaluation harnesses used to benchmark AI agents. Participants will review tasks and harnesses and provide feedback on realism and usefulness.

We are seeking practicing software engineers with hands-on experience building and testing complex systems, including writing test suites, evaluation harnesses, or performing comprehensive code reviews. Sessions are conducted via guided conversations and screen-sharing.

Qualifications

• Practicing software engineers with experience building and testing complex systems.
• Direct experience writing test suites, evaluation harnesses, or performing code reviews.
• Background in backend, full-stack, ML engineering, or software architecture is preferred.

Responsibilities

• Review coding tasks and evaluation harnesses for realism.
• Verify logic, test cases, and structure of the environments provided.
• Provide technical feedback via guided conversation and screen-sharing.

Skills

Software engineering

Test suite development

Code reviews

Evaluation harnesses

Backend development

Education

Bachelor's degree in Computer Science or related field

Tools

Git

CI/CD

Job description

What We're Researching

We're running a paid study on the coding environments and programming tasks used to benchmark artificial intelligence agents. Creating robust evaluation harnesses ensures that AI models are tested against realistic software engineering scenarios. This work directly feeds into improving how autonomous agents handle complex coding objectives.

How It Works

During this remote session, you will review a series of coding tasks and their corresponding evaluation harnesses. You will assess whether the programming challenges accurately reflect real-world software engineering problems. We will ask you to verify the logic, test cases, and overall structure of the environments provided. Your technical feedback will be captured through a guided conversation and screen-sharing exercises.

Who This Is For

We are looking for practicing software engineers with strong backgrounds in building and testing complex systems. Candidates should have direct experience writing test suites, evaluation harnesses, or comprehensive code reviews. We welcome backend engineers, full-stack developers, machine learning engineers, and software architects.