AI Evaluation Architect - Project-Based Code Agent Testing

Mindrift; New York (NY); On-site; USD 41,000 - 69,000; Full time

Benefits offered by this job

Flexible schedule

Project-based work

Job summary

Mindrift is building a dataset to evaluate AI coding agents by creating realistic developer environments, tasks, and tests. You will guide and evaluate the agent’s work, ensuring the tasks have valid multiple solutions and fair criteria.

This contract-based role offers up to $50/hr, with ~20 hours per task and flexible scheduling, requiring 5+ years in software development with a core stack including Python, FastAPI, React, Docker, Postgres, Kafka, Redis and testing experience.

Qualifications

• 5+ years in software development with a strong backend/frontend track record.
• Experience building or evaluating development environments and tests.
• Ability to design tasks with multiple valid solutions and clear evaluation criteria.

Responsibilities

• Create realistic developer environments and contexts for AI evaluation.
• Design tasks, define success criteria, and ensure solvability for AI agents.
• Write tests that validate agent solutions across valid approaches.
• Iterate on tasks based on QA feedback to keep evaluations fair and robust.

Skills

Python

FastAPI

JavaScript/TypeScript

React

Docker

Postgres

Kafka

Redis

Testing

Job description

Mindrift is building a dataset to evaluate AI coding agents by creating realistic developer environments, tasks, and tests. You will guide and evaluate the agent’s work, ensuring the tasks have valid multiple solutions and fair criteria.

This contract-based role offers up to $50/hr, with ~20 hours per task and flexible scheduling, requiring 5+ years in software development with a core stack including Python, FastAPI, React, Docker, Postgres, Kafka, Redis and testing experience.