Remote AI Agent Evaluation Engineer

YO AI Labs; New York (NY); Remote; USD 46,000 - 110,000; Part time

Job summary

YO AI Labs is seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Qualifications

• Experience creating reinforcement learning environments for software engineering tasks.
• Ability to design reproducible environments and deterministic verification.
• Experience with codebase refactoring and performance optimization recommended.

Responsibilities

• Create reinforcement learning environments for software engineering tasks.
• Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
• Build deterministic verification systems and golden reference solutions.
• Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
• Develop realistic, reproducible software engineering scenarios.
• Document solutions and provide clear technical feedback.

Skills

Python 3

Java

Rust

C++

TypeScript

Algorithms & data structures

Bug fixing & debugging

Feature implementation & refactoring

Performance optimization

Communication

Attention to detail

Distributed codebases

Job description

YO AI Labs is seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.