AI Code-Evaluation Architect

Mindrift; Singapore; On-site; SGD 34,000 - 48,000; Full time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

You’ll design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome.

Qualifications

• 3+ years of production software development in one backend stack (Python, Go, Node.js, Java, or Rust).
• Python + pytest fluency is required; the task harness is pytest-based even if the broken app is in another language.
• Docker authoring with reproducible Dockerfiles, pinned dependencies, multi-stage builds when needed, non-root user.
• Linux and Bash proficiency; debugging inside containers with tools like strace, lsof, and journalctl.
• AI coding agent experience—Cite a time you caught an AI misstep and how you addressed it.
• English — B2+ written.

Responsibilities

• Design coding tasks as self-contained Docker environments with a broken piece of software.
• Create a reproducible Docker environment with pinned dependencies.
• Write a deterministic pytest that verifies outcomes without leaking the fix.
• Draft an instruction.md that reads like a Jira ticket for a developer.
• Provide a reference solve.sh proving the task is solvable.
• Calibrate difficulty so current state-of-the-art agents solve the task 20–60% of the time.
• Iterate based on expert QA feedback and later review other authors’ tasks as a QA reviewer.

Skills

Python

pytest

Docker

Linux

AI coding agent experience

Tools

Docker

pytest

strace

lsof

journalctl

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

You’ll design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome.