Senior Python Engineer — AI Evaluation & Task Design

Socket.dev; Iowa (LA); Hybrid; USD 83,000 - 276,000; Full time

Job summary

Mindrift engages in project-based AI evaluation, designing tasks that simulate real developer environments with codebases, infrastructure, and context. You will define what counts as a solution and write tests to verify agent submissions, iterating based on QA feedback.

This is a flexible, contract-like role, not permanent employment, with compensation per accepted task and rates up to $200 per hour. We seek engineers with 8+ years of software experience in Python (FastAPI), TypeScript/React,

Qualifications

• 8+ years in software development.
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+

Responsibilities

• Create challenging tasks and evaluation criteria within realistic simulated environments.
• Build realistic developer environments with codebases, infrastructure, and context.
• Design tasks from intermediate states of these environments and craft what counts as solved.
• Write tests that verify agent solutions, accepting all valid approaches and rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback to ensure fairness and robustness.

Skills

Python

TypeScript

React

Docker

Postgres

Kafka

Redis

Testing

English proficiency

Tools

FastAPI

Job description

Mindrift engages in project-based AI evaluation, designing tasks that simulate real developer environments with codebases, infrastructure, and context. You will define what counts as a solution and write tests to verify agent submissions, iterating based on QA feedback.

This is a flexible, contract-like role, not permanent employment, with compensation per accepted task and rates up to $200 per hour. We seek engineers with 8+ years of software experience in Python (FastAPI), TypeScript/React,