AI Coding Task Designer & Evaluation Specialist

Mindrift; New York (NY); On-site; USD 34,440,000 - 48,216,000; Full time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. You’ll design coding tasks as self-contained Docker environments with broken code; an AI agent attempts the fix; automated tests verify the outcome.

Your deliverable is the full task package: broken code, tests, instructions, and a reference solution proving the task is solvable.

Qualifications

• 3+ years of production software development in a backend stack (Python/Go/Node.js/Java/Rust).
• Python + pytest fluency required.
• Docker authoring with reproducible images and pinned dependencies.
• Linux & Bash proficiency for debugging inside containers.

Responsibilities

• Design coding tasks as self-contained Docker environments with broken code and automated tests.
• Invent realistic developer scenarios (bugs, ETL issues, missing features).
• Build reproducible Docker environments with pinned dependencies.
• Write a deterministic pytest that verifies outcomes without revealing fixes.
• Create instruction.md that reads like a Jira ticket and a reference solve.sh.
• Calibrate difficulty so agents solve 20–60% of tasks; iterate based on QA feedback.

Skills

Backend development

Python + pytest

Docker authoring

Linux & Bash

AI coding agent experience

English B2+

Tools

Docker

pytest

strace

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. You’ll design coding tasks as self-contained Docker environments with broken code; an AI agent attempts the fix; automated tests verify the outcome.

Your deliverable is the full task package: broken code, tests, instructions, and a reference solution proving the task is solvable.