AI Task Architect & Evaluation Engineer

Mindrift; South Carolina; On-site; USD 41,000 - 69,000; Part time

Benefits offered by this job

Competitive compensation

Flexible scheduling

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. This role involves building realistic developer environments, crafting tasks, and writing tests to verify agent solutions.

You will work on an evaluation dataset for AI coding agents, set clear success criteria, and iterate tasks based on QA feedback. Compensation up to $50/hr, with flexible scheduling.

Qualifications

• 5+ years in software development.
• Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Build realistic developer environments for evaluation datasets.
• Design tasks from intermediate states of environments and define what 'solved' means.
• Write tests to verify agent solutions and ensure multiple valid approaches are accepted.
• Iterate on tasks and tests based on QA feedback to ensure fairness and robustness.

Skills

Software development experience

Python

JavaScript/TypeScript

Testing (functional/integration)

English (B2+)

Problem solving

Tools

FastAPI

React

Docker

Postgres

Kafka

Redis

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. This role involves building realistic developer environments, crafting tasks, and writing tests to verify agent solutions.

You will work on an evaluation dataset for AI coding agents, set clear success criteria, and iterate tasks based on QA feedback. Compensation up to $50/hr, with flexible scheduling.