Senior Python Engineer — AI Task Designer & Evaluator

Socket.dev; Pennsylvania; Hybrid; USD 40,000 - 90,000; Full time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves: Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history

Qualifications

• 8+ years of software development experience with modern stacks.
• Experience writing functional and integration tests.
• English proficiency at B2+ level.

Responsibilities

• Build realistic developer environments – a virtual company with codebase, infra, and context.
• Design tasks from intermediate states of environments – define what counts as solved.
• Write tests that verify agent solutions – accept all valid approaches and reject incorrect ones.
• Iterate on tasks and tests based on QA feedback – refine until evaluation is fair and robust.

Skills

8+ years in software development

Core stack knowledge

Test writing skills (functional, tests

English proficiency (B2+)

Tools

Python (FastAPI)

JavaScript/TypeScript (React)

Docker

Postgres

Kafka

Redis

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves: Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history