AI Agent Evaluation Engineer - Remote

YO AI Labs; Boston (MA); Remote; USD 42,000 - 65,000; Full time

Job summary

YO AI Labs is seeking a Senior Software Engineer contractor to help build reinforcement learning environments evaluating AI models on software tasks. The role focuses on creating reproducible environments, deterministic verification, and reference solutions for code challenges.

Responsibilities include designing tasks such as bug fixing, feature development, refactoring, and performance optimization, while evaluating agents' reasoning with MCP tools. Remote, ~15 hours/week.

Qualifications

• Proficient in Python 3, Java, Rust, C++ or TypeScript.
• Experience debugging complex software issues and bug fixing.
• Strong problem-solving, attention to detail and communication.

Responsibilities

• Create reinforcement learning environments for software tasks.
• Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
• Build deterministic verification systems and golden reference solutions.
• Evaluate AI agents on codebases using MCP tools.
• Develop realistic, reproducible software engineering scenarios.

Skills

Python 3

Java

Rust

C++

TypeScript

Job description

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

• Create reinforcement learning environments for software engineering tasks.
• Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
• Build deterministic verification systems and golden reference solutions.
• Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
• Develop realistic, reproducible software engineering scenarios.
• Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
• Document solutions and provide clear technical feedback.

Required Skills

• Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
• Strong understanding of algorithms and data structures.
• Experience with bug fixing and debugging complex software issues.
• Proven experience in feature implementation and codebase refactoring.
• Strong knowledge of performance optimization and tuning.
• Excellent written and verbal communication.
• Strong attention to detail.

Preferred Qualifications

• Experience working with large-scale or distributed codebases.
• Familiarity with AI/ML systems is a plus but not required.
• Experience with rigorous code reviews and software engineering best practices.
• Experience working effectively in remote or cross-functional teams.

Hiring Process

1. Submit an application and screening questions.
2. Complete an AI interview (~30 minutes).
3. Complete a technical assessment, if required.
4. Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.