AI Evaluation Engineer: Build Real-World Coding Challenges

Mindrift; Doha; On-site; QAR 207,000 - 295,000; Full time

Benefits offered by this job

Flexible schedule

Competitive hourly rate

Job summary

Mindrift is assembling a dataset and evaluation framework to test AI coding agents. You will construct realistic developer environments, including codebases, infrastructure, and contextual artifacts such as tickets and docs, to simulate a believable history.

You will design tasks from intermediate states, define solvable criteria for AI agents, and craft tests that tolerate multiple valid approaches while rejecting incorrect ones.

Qualifications

• 5+ years in software development
• Core stack: Python (Fast API), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
• Experience writing tests (functional, integration)
• English proficiency - B2+

Responsibilities

• Build realistic developer environments: a virtual company with codebase, infrastructure, and context
• Design tasks from intermediate states of these environments and define what solved means
• Write tests that verify agent solutions, accepting all valid approaches and rejecting incorrect ones
• Iterate on tasks and tests based on QA feedback to ensure fairness and robustness

Skills

Python

JavaScript/TypeScript

React

Docker

PostgreSQL

Kafka

Redis

Testing

English proficiency

Tools

Docker

PostgreSQL

Kafka

Redis

Job description

Mindrift is assembling a dataset and evaluation framework to test AI coding agents. You will construct realistic developer environments, including codebases, infrastructure, and contextual artifacts such as tickets and docs, to simulate a believable history.

You will design tasks from intermediate states, define solvable criteria for AI agents, and craft tests that tolerate multiple valid approaches while rejecting incorrect ones.