Senior Software Engineer - AI Evaluation & Task Design

Dorado; United States; Remote; USD 44,083 - 55,104; Full time

Job summary

Dorado seeks a contributor in the United States to help build a dataset for evaluating AI coding agents. You will design challenging tasks and evaluation criteria within realistic simulated environments—creating developer environments, tickets, docs, and conversations that form a believable history.

You will write tests that accept diverse valid solutions, iterate on tasks based on QA feedback, and participate end-to-end from applying to getting paid.

Qualifications

• 5+ years of software development experience.
• Core stack includes Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Build realistic developer environments and a believable development history.
• Design tasks from intermediate states of environments and define what counts as solved.
• Write tests that accept all valid solutions and reject incorrect ones.
• Iterate on tasks and tests based on QA feedback to keep evaluation fair and robust.

Skills

5+ years in software development

English proficiency

Test writing

Tools

Python

FastAPI

JavaScript/TypeScript

React

Docker

PostgreSQL

Kafka

Redis

Job description

Dorado seeks a contributor in the United States to help build a dataset for evaluating AI coding agents. You will design challenging tasks and evaluation criteria within realistic simulated environments—creating developer environments, tickets, docs, and conversations that form a believable history.

You will write tests that accept diverse valid solutions, iterate on tasks based on QA feedback, and participate end-to-end from applying to getting paid.