Senior AI Evaluation Engineer – Agent Testing & Tasks

Dorado; United States; Remote; USD 55,104 - 82,656; Part time

Job summary

Mindrift seeks contributors to build a dataset for evaluating AI coding agents. You will create tasks and evaluation criteria within realistic simulated environments, including a virtual codebase, infrastructure, tickets, and conversations to form a believable development history.

You will design prompts, define success criteria, and write tests that accept diverse correct solutions while rejecting incorrect ones. This is a project-based engagement rather than permanent employment.

Qualifications

• 5+ years in software development.
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Build realistic developer environments — a virtual company with codebase, infrastructure, and context (tickets, docs, conversations).
• Design tasks from intermediate states of these environments — craft the prompt, define what “solved” means, and ensure solvable by an AI agent.
• Write tests that verify agent solutions — accept all valid approaches and reject incorrect ones, not too strict or lenient.
• Iterate on tasks and tests based on QA feedback — review agent solutions, analyze failures, and refine until evaluation is fair and robust.

Skills

5+ years in software development

Python

JavaScript/TypeScript

Docker

Postgres

Kafka

Redis

Testing (functional, integration)

English proficiency (B2+)

Tools

FastAPI

React

Docker

PostgreSQL

Kafka

Redis

Job description

Mindrift seeks contributors to build a dataset for evaluating AI coding agents. You will create tasks and evaluation criteria within realistic simulated environments, including a virtual codebase, infrastructure, tickets, and conversations to form a believable development history.

You will design prompts, define success criteria, and write tests that accept diverse correct solutions while rejecting incorrect ones. This is a project-based engagement rather than permanent employment.