Senior Software Engineer - AI Task Designer (Project-Based)

Dorado; United States; Remote; USD 34,440 - 55,104; Full time

Job summary

Mindrift is building a dataset to evaluate AI coding agents by creating realistic developer scenarios, environments, and evaluation criteria. You’ll craft tasks from simulated states, write tests, and iteratively refine the tasks based on QA feedback.

This project-based role focuses on guiding and evaluating agent solutions rather than writing code from scratch. Compensation scales with the level and pace, up to $40/hr equivalent.

Qualifications

• 5+ years in software development.
• Proficient in Python (FastAPI) and JavaScript/TypeScript (React).
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Create challenging tasks and evaluation criteria within realistic simulated environments.
• Design tasks from intermediate states of environments and define what constitutes a solved state.
• Write tests that accept all valid approaches and reject incorrect ones, neither too strict nor too lenient.
• Iterate on tasks and tests based on QA feedback to ensure fairness and robustness.

Skills

Python

JavaScript/TypeScript

Docker

Postgres

Kafka

Redis

Testing

English (B2+)

Tools

Docker

PostgreSQL

Kafka

Redis

Job description

Mindrift is building a dataset to evaluate AI coding agents by creating realistic developer scenarios, environments, and evaluation criteria. You’ll craft tasks from simulated states, write tests, and iteratively refine the tasks based on QA feedback.

This project-based role focuses on guiding and evaluating agent solutions rather than writing code from scratch. Compensation scales with the level and pace, up to $40/hr equivalent.