AI Evaluation Engineer: Coding Task Architect

United States Digital Space LLC; United States; Remote; USD 55,000 - 69,000; Full time

Job summary

United States Digital Space LLC is building a dataset to evaluate AI coding agents by creating realistic developer environments with codebases, infrastructure, tickets, docs, and a believable development history. You will design tasks from intermediate states and craft what a solution must achieve to be considered solved.

The role emphasizes writing tests that accept all valid approaches while rejecting incorrect ones, and iterating on tasks based on QA feedback to keep evaluations fair and

Qualifications

• 5+ years in software development.
• Proficiency with Python (FastAPI) and JavaScript/TypeScript (React).
• Experience with Docker, PostgreSQL, Kafka, and Redis.
• Experience writing tests (functional, integration).
• English proficiency - B2+.

Responsibilities

• Build realistic developer environments—a virtual company with codebase, infra, tickets and docs.
• Design tasks from intermediate states of these environments to be solvable by AI agents.
• Write tests that verify agent solutions and balance strictness with correctness.
• Iterate on tasks and tests based on QA feedback to ensure fairness and robustness.

Skills

Python

JavaScript

TypeScript

React

Docker

PostgreSQL

Kafka

Redis

Testing

Tools

Docker

PostgreSQL

Kafka

Redis

FastAPI

React

Job description

United States Digital Space LLC is building a dataset to evaluate AI coding agents by creating realistic developer environments with codebases, infrastructure, tickets, docs, and a believable development history. You will design tasks from intermediate states and craft what a solution must achieve to be considered solved.

The role emphasizes writing tests that accept all valid approaches while rejecting incorrect ones, and iterating on tasks based on QA feedback to keep evaluations fair and