AI Task Design & Evaluation Engineer

Mindrift; Michigan; On-site; USD 27,552,000 - 48,216,000; Full time

Job summary

Mindrift seeks an experienced engineer to design complex developer tasks for evaluating AI coding agents. You will craft self-contained Docker environments with a broken piece of software, and publish full task packages including tests and references showing solvability.

You’ll deliver instructions read like Jira tickets, write a deterministic pytest, and calibrate task difficulty so current agents solve 20–60% of the time, iterating with QA feedback to raise quality over time.

Qualifications

• 3+ years of production software development in one backend stack (Python, Go, Node.js, Java, or Rust).
• Python + pytest fluency required regardless of primary stack.
• Docker authoring with reproducible Dockerfiles and pinned dependencies.

Responsibilities

• Invent a realistic developer scenario — a real bug, a broken ETL, or a missing feature.
• Build a reproducible Docker environment with pinned dependencies.
• Write a pytest that verifies outcomes deterministically, non-flaky and does not leak the fix.
• Write an instruction.md that reads like a Jira ticket a developer would receive.
• Write a reference solve.sh proving the task is solvable.

Skills

Python

pytest

Docker

Linux

Bash

AI coding agent experience

English proficiency

Tools

Docker

pytest

Job description

Mindrift seeks an experienced engineer to design complex developer tasks for evaluating AI coding agents. You will craft self-contained Docker environments with a broken piece of software, and publish full task packages including tests and references showing solvability.

You’ll deliver instructions read like Jira tickets, write a deterministic pytest, and calibrate task difficulty so current agents solve 20–60% of the time, iterating with QA feedback to raise quality over time.