Senior Python Engineer – AI Task Designer & Evaluator

Socket.dev; North Carolina; Hybrid; USD 138,000 - 276,000; Full time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. This is a project-based opportunity, not permanent employment, with paid tasks and a potential rate up to $200/hr.

Responsibilities include building realistic developer environments, crafting tasks from intermediate states, writing tests to verify solutions, and iterating tasks based on QA feedback.

Qualifications

• 8+ years in software development.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations).
• Design tasks from intermediate states of these environments - craft the prompt, define what 'solved' means, and ensure the task is solvable by an AI agent.
• Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient.
• Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust.

Tools

Python (FastAPI)

JavaScript/TypeScript (React)

Docker

Postgres

Kafka

Redis

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. This is a project-based opportunity, not permanent employment, with paid tasks and a potential rate up to $200/hr.

Responsibilities include building realistic developer environments, crafting tasks from intermediate states, writing tests to verify solutions, and iterating tasks based on QA feedback.