Senior Python Engineer - AI Evaluation Task Designer (Contract)

Mindrift; Manila; On-site; PHP 7,630,000 - 12,716,000; Full time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This role centers on building realistic developer environments, crafting tasks from intermediate states, and designing precise evaluation criteria that reflect real-world coding work.

You will write tests that accept all valid approaches and reject incorrect ones, iterate on tasks with QA feedback, and help ensure the evaluation remains fair,

Qualifications

• 8+ years in software development.
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Build realistic developer environments - a virtual company with codebase, infrastructure, and context that forms a believable development history.
• Design tasks from intermediate states of these environments - craft the prompt, define what is solved, and ensure solvability by an AI agent.
• Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones.
• Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until evaluation is fair and robust.

Skills

8+ years in software development

English proficiency (B2+)

Tools

Python (FastAPI)

JavaScript/TypeScript (React)

Docker

Postgres

Kafka

Redis

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This role centers on building realistic developer environments, crafting tasks from intermediate states, and designing precise evaluation criteria that reflect real-world coding work.

You will write tests that accept all valid approaches and reject incorrect ones, iterate on tasks with QA feedback, and help ensure the evaluation remains fair,