Remote Freelance Senior Python Engineer — AI Evaluation

Toloka AI AG (Mindrift); United States; On-site; USD 96,000 - 165,000; Part time

Job summary

Toloka AI AG (Mindrift) is seeking a Senior Python Engineer for remote, part-time freelance work to build and evaluate tasks for AI coding agents, create realistic developer environments, and define evaluation criteria to assess agent performance.

The role requires Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka and Redis, plus experience writing functional and integration tests. English proficiency at B2 level or higher is essential.

Qualifications

• 8+ years of experience in software development.
• Proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
• Experience in writing functional and integration tests.
• English proficiency at a B2 level or higher.

Responsibilities

• Build realistic developer environments that simulate a virtual company with a codebase and context
• Design tasks and evaluation criteria to ensure AI agents can effectively solve developer challenges
• Write and iterate on tests that verify agent solutions, refining them based on quality assurance feedback

Skills

Python

JavaScript/TypeScript

Docker

Postgres

Kafka

Redis

Testing

Job description

Toloka AI AG (Mindrift) is seeking a Senior Python Engineer for remote, part-time freelance work to build and evaluate tasks for AI coding agents, create realistic developer environments, and define evaluation criteria to assess agent performance.

The role requires Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka and Redis, plus experience writing functional and integration tests. English proficiency at B2 level or higher is essential.