Agent Evaluation Engineer

Employment

Doha

Remote

QAR 207,000 - 295,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Employment seeks experienced software developers and testers to help build a realistic developer environment and evaluation tasks for AI coding agents. You will craft prompts, define success criteria, and write tests that accept multiple valid approaches.

This project uses Python/FastAPI, TypeScript/React, Docker, PostgreSQL, Kafka and Redis; collaboration with a remote team is required and English proficiency (B2+) is essential.

Qualifications

  • 5+ years of professional software development experience.
  • Strong hands-on with Python (FastAPI) and JS/TS (React).
  • Experience with Docker, PostgreSQL, Kafka and Redis.
  • Proven testing experience (functional/integration tests) and English proficiency (B2+).

Responsibilities

  • Create realistic developer environments including codebase, infra, tickets and docs.
  • Design tasks from intermediate states and define what ‘solved’ means.
  • Write tests that accept multiple valid approaches and reject incorrect ones.
  • Iterate on tasks based on QA feedback to ensure fairness and robustness.

Skills

5+ years software development
Python (FastAPI)
JavaScript/TypeScript (React)
Docker
PostgreSQL
Kafka
Redis
Testing experience (functional/integr.
English proficiency (B2+)

Tools

Docker
PostgreSQL
Kafka
Redis
React
FastAPI

Job description

Industry Information Technology and Services

Please submit your CV in English and indicate your level of English proficiency.

What This Opportunity Involves

We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:
  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What This Is NOT
  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate
What We Look For
  • 5+ years in software development Core stack: Python (Fast API), Java Script/Type Script (React), Docker, Postgres, Kafka, Redis Experience writing tests (functional, integration) English proficiency - B2+
Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How It Works
  • Apply
  • Pass qualification(s)
  • Join a project
  • Complete tasks
  • Get paid
Effort estimate

Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation

Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at :20 hours each; you set your own schedule.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Coding Agent Evaluator — Design & Validate Tasks
AI Coding Agent Evaluator — Design & Validate Tasks

Employment • Doha

Remote
QAR 207,000 - 295,000
Senior AI Evals Engineer
Senior AI Evals Engineer

Alpha10x • Doha

On-site
QAR 360,000 - 600,000
Freelance Agent Evaluation Engineer
Freelance Agent Evaluation Engineer

Employment • Doha

On-site
QAR 207,000 - 295,000
Project-based engagement
Senior AI Engineer
Senior AI Engineer

Alpha10x • Doha

On-site
QAR 300,000 - 540,000
Forward Deployed Engineer - Doha
Forward Deployed Engineer - Doha

present of AI • Doha

On-site
QAR 70,000 - 110,000
Arabic Language Expert
Arabic Language Expert

Employment • Doha

Remote
QAR 125,000 - 201,000
Senior QA Engineer - AI Products (6 Months Contract)
Senior QA Engineer - AI Products (6 Months Contract)

Arabic.AI • Doha

On-site
QAR 180,000 - 240,000
Staff / Principal Engineer - AI-Native Engineering
Staff / Principal Engineer - AI-Native Engineering

TechTree • Doha

Hybrid
QAR 600,000 - 1,000,000
Hybrid work model
Senior AI Engineer
Senior AI Engineer

Cplusoft • Doha

On-site
QAR 400,000 - 800,000
Hindi Speakers (Freelance - Remote)
Hindi Speakers (Freelance - Remote)

Prolific • Doha

Remote
QAR 75,000 - 105,000
Top locations