Software Engineer - AI Agent Evaluation

aitrainer; New York, Northern (NY, KY); Hybrid; USD 57,000 - 81,000; Full time

Job summary

Mindrift connects specialists with project-based AI opportunities for leading tech companies. You will create challenging tasks and evaluation criteria within realistic simulated environments, forming believable development histories and ensuring tasks are solvable by AI agents.

The role emphasizes writing tests that accept multiple correct solutions and requires 5+ years in software development plus strong Python/JS stack experience.

Qualifications

• 5+ years in software development.
• Proficiency with Python (FastAPI) and JS/TS (React).
• Experience writing tests (functional, integration).
• English proficiency – B2+.
• Familiarity with Docker, Postgres, Kafka, Redis.

Responsibilities

• Build realistic developer environments – a virtual company with codebase, infrastructure, and context.
• Design tasks from intermediate states of these environments – craft prompts and success criteria.
• Write tests that verify agent solutions and accept all valid approaches.
• Iterate on tasks and tests based on QA feedback and refine until evaluation is fair and robust.

Skills

Python (FastAPI)

JavaScript/TypeScript (React)

Docker

Postgres

Kafka

Redis

Testing (functional, integration)

English proficiency (B2+)

Job description

Mindrift connects specialists with project-based AI opportunities for leading tech companies. You will create challenging tasks and evaluation criteria within realistic simulated environments, forming believable development histories and ensuring tasks are solvable by AI agents.

The role emphasizes writing tests that accept multiple correct solutions and requires 5+ years in software development plus strong Python/JS stack experience.