Senior Python Engineer - AI Evaluation & Testing

Dorado; United States; Remote; USD 165,312 - 275,520; Full time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies. This role focuses on building a dataset to evaluate AI coding agents and how well models handle real-world developer tasks.

The work is project-based, not permanent employment, with responsibilities including creating believable environments, crafting tasks, writing tests, and iterating based on QA feedback. Strong software background and English proficiency are required.

Qualifications

• 8+ years in software development.
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Build realistic developer environments: a virtual company with a codebase, infrastructure, and context that forms a believable development history.
• Design tasks from intermediate states of these environments: craft the prompt, define what "solved" means, and ensure solvable by an AI agent.
• Write tests that verify agent solutions: accept all valid approaches and reject incorrect ones, neither too strict nor too lenient.
• Iterate on tasks and tests based on QA feedback: review agent solutions, analyze failures, and refine until the evaluation is fair and robust.

Skills

Software engineering experience

English proficiency

Testing experience

AI task design

Tools

Python

FastAPI

JavaScript/TypeScript

React

Docker

Postgres

Kafka

Redis

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies. This role focuses on building a dataset to evaluate AI coding agents and how well models handle real-world developer tasks.

The work is project-based, not permanent employment, with responsibilities including creating believable environments, crafting tasks, writing tests, and iterating based on QA feedback. Strong software background and English proficiency are required.