AI Benchmark Engineer: Backend Task Design & Evaluation

Mindrift; Glasgow; On-site; GBP 21,000 - 35,000; Full time

Job summary

Mindrift in Glasgow, Scotland is seeking a Software Engineering Evaluation Specialist to design AI-task challenges. You will craft self-contained Docker environments containing broken code, tests, instructions, and a reference solution.

You’ll ensure reproducibility and deterministic verification with pytest, while producing clear Jira-style tickets and a solvable reference script. The role emphasizes project-based work rather than permanent employment, with on-demand contributions and

Qualifications

• 3+ years of production software development in a backend stack.
• Fluency in Python and pytest required regardless of primary language.
• Experience with Docker-based task environments and Linux debugging.

Responsibilities

• Design reproducible Docker environments with pinned dependencies.
• Write deterministic pytest tests that verify outcomes without leaking fixes.
• Create instruction.md styled as a Jira ticket for developers.
• Provide a reference solve.sh proving task solvability.
• Calibrate task difficulty for 20–60% success by current AI agents.
• Iterate based on QA reviewer feedback; later act as QA reviewer for others.

Skills

Python

pytest

Docker

Linux

Bash

AI coding agent experience

Tools

Docker

pytest

CI/CD

Job description

Mindrift in Glasgow, Scotland is seeking a Software Engineering Evaluation Specialist to design AI-task challenges. You will craft self-contained Docker environments containing broken code, tests, instructions, and a reference solution.

You’ll ensure reproducibility and deterministic verification with pytest, while producing clear Jira-style tickets and a solvable reference script. The role emphasizes project-based work rather than permanent employment, with on-demand contributions and