AI Agent Evaluation Engineer — Design & Test Coding Tasks

Mindrift; New York (NY); Remote; USD 41,000 - 69,000; Full time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. You will help build a dataset to evaluate AI coding agents in realistic environments.

Work involves creating tasks, defining success criteria, and writing robust tests that accept diverse correct solutions. This is a contract, project-based role with variable hours and compensation.

Qualifications

• 5+ years in software development.
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, PostgreSQL, Kafka, Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations).
• Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent.
• Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient.
• Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust.

Skills

Python

JavaScript/TypeScript

React

Testing

Tools

Docker

PostgreSQL

Kafka

Redis

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. You will help build a dataset to evaluate AI coding agents in realistic environments.

Work involves creating tasks, defining success criteria, and writing robust tests that accept diverse correct solutions. This is a contract, project-based role with variable hours and compensation.