Senior Python Engineer for AI Task Design & Testing

Mindrift; Singapore; On-site; SGD 106,000 - 352,000; Full time

Job summary

Mindrift invites experts to contribute to a project-based AI evaluation effort. You will craft realistic tasks and environments that simulate a development company, with tests and criteria to measure agent performance on real-world coding tasks.

You design prompts, define success, and validate multiple valid approaches while rejecting incorrect ones. This is a flexible, non-permanent engagement focused on quality evaluation.

Qualifications

• 8+ years in software development.
• Core stack includes Python (FastAPI) and JavaScript/TypeScript (React).
• Strong experience writing tests (functional, integration).
• English proficiency at least B2+.

Responsibilities

• Build realistic developer environments: a virtual company with codebase, infra, tickets, docs, and conversations.
• Design tasks from intermediate states; craft prompt, define success, and ensure solvability by an AI agent.
• Write tests that verify agent solutions, accepting all valid approaches and rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback until evaluation is fair and robust.

Skills

8+ years experience

English proficiency (B2+)

Experience writing tests

Tools

Python (FastAPI)

JavaScript/TypeScript (React)

Docker

Postgres

Kafka

Redis

Job description

Mindrift invites experts to contribute to a project-based AI evaluation effort. You will craft realistic tasks and environments that simulate a development company, with tests and criteria to measure agent performance on real-world coding tasks.

You design prompts, define success, and validate multiple valid approaches while rejecting incorrect ones. This is a flexible, non-permanent engagement focused on quality evaluation.