Senior Python Engineer — AI Task Designer & Evaluator

Dorado; United States; Remote; USD 165,312 - 275,520; Full time

Job summary

Mindrift is offering a project-based opportunity to craft and evaluate AI coding tasks. You will build a realistic developer environment—a virtual company with a codebase, infrastructure, tickets, docs, and conversations—to test AI agents.

You design tasks from intermediate states, define what constitutes a solution, and write tests that accept all valid approaches while rejecting incorrect ones. English proficiency required; 8+ years of software development preferred.

Qualifications

• 8+ years of software development experience.
• Proficiency with Python (FastAPI) and JavaScript/TypeScript (React).
• Experience writing functional and integration tests.
• English proficiency at B2+ level.

Responsibilities

• Build realistic developer environments: a virtual company with codebase, infra, tickets, docs, and conversations.
• Design tasks from intermediate states of these environments: craft the prompt, define what "solved" means, and ensure solvability by an AI agent.
• Write tests that verify agent solutions: accept all valid approaches and reject incorrect ones, neither too strict nor too lenient.
• Iterate on tasks and tests based on QA feedback: review agent solutions, analyze failures, and refine until evaluation is fair and robust.

Skills

Software development experience

Python

JavaScript/TypeScript

Testing

English B2+

Tools

Docker

Postgres

Kafka

Redis

Job description

Mindrift is offering a project-based opportunity to craft and evaluate AI coding tasks. You will build a realistic developer environment—a virtual company with a codebase, infrastructure, tickets, docs, and conversations—to test AI agents.

You design tasks from intermediate states, define what constitutes a solution, and write tests that accept all valid approaches while rejecting incorrect ones. English proficiency required; 8+ years of software development preferred.