Senior AI Agent Evaluation Engineer

Worky; United States; Remote; USD 55,000 - 69,000; Full time

Job summary

Mindrift is creating a dataset to evaluate AI coding agents by simulating real-world developer tasks. You will guide and evaluate the agent-written code, not write code from scratch, and help design tests that challenge Frontier models in coding scenarios.

This project-based role offers flexible scheduling and compensation up to $50/hr, depending on level and pace, with roughly 20 hours per task.

Qualifications

• 5+ years in software development.
• Strong stack knowledge: Python (FastAPI), JS/TS (React), Docker, Postgres, Kafka, Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Assist in building a dataset to evaluate AI coding agents.
• Guide and evaluate agent-generated code rather than writing code from scratch.
• Develop tests to validate model outputs and assess real-world developer tasks.

Skills

Software development experience

Testing experience

English proficiency

Tools

Python (FastAPI)

JavaScript/TypeScript (React)

Docker

Postgres

Kafka

Redis

Job description

Mindrift is creating a dataset to evaluate AI coding agents by simulating real-world developer tasks. You will guide and evaluate the agent-written code, not write code from scratch, and help design tests that challenge Frontier models in coding scenarios.

This project-based role offers flexible scheduling and compensation up to $50/hr, depending on level and pace, with roughly 20 hours per task.