Remote Senior AI Agent Evaluation Engineer

YO AI Labs; Boston (MA); Remote; USD 42,000 - 65,000; Full time

Job summary

YO AI Labs is seeking a Senior Software Engineer contractor to help build reinforcement learning environments evaluating AI models on software tasks. The role focuses on creating reproducible environments, deterministic verification, and reference solutions for code challenges.

Responsibilities include designing tasks such as bug fixing, feature development, refactoring, and performance optimization, while evaluating agents' reasoning with MCP tools. Remote, ~15 hours/week.

Qualifications

• Proficient in Python 3, Java, Rust, C++ or TypeScript.
• Experience debugging complex software issues and bug fixing.
• Strong problem-solving, attention to detail and communication.

Responsibilities

• Create reinforcement learning environments for software tasks.
• Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
• Build deterministic verification systems and golden reference solutions.
• Evaluate AI agents on codebases using MCP tools.
• Develop realistic, reproducible software engineering scenarios.

Skills

Python 3

Java

Rust

C++

TypeScript

Job description

YO AI Labs is seeking a Senior Software Engineer contractor to help build reinforcement learning environments evaluating AI models on software tasks. The role focuses on creating reproducible environments, deterministic verification, and reference solutions for code challenges.

Responsibilities include designing tasks such as bug fixing, feature development, refactoring, and performance optimization, while evaluating agents' reasoning with MCP tools. Remote, ~15 hours/week.