AI Agent Evaluator - Remote

YO AI Labs; New York (NY); Remote; USD 55,000 - 83,000; Part time

Job summary

YO AI Labs seeks detail-oriented professionals to evaluate AI agents performing everyday tasks. You will test two agents side by side, assess their performance, and provide actionable feedback to improve reliability, usability, and user satisfaction.

Based in the United States, you will document results with screen recordings, rate task completion and quality, and decide whether to approve or stop agents at key steps. The project kicks off soon with onboarding prior to start.

Qualifications

• Based in the United States with a bachelor's degree.
• 2–3 years of professional experience in related fields.
• Active personal AI account and a smartphone capable of screen recording.
• Regular use of AI tools in daily life.

Responsibilities

• Test AI agents on everyday tasks, including shopping, travel, bookings, bill payments, email follow-ups, and forms.
• Approve or stop agents at key steps.
• Screen-record each testing session.
• Score task completion, quality, control issues, user satisfaction, and time.
• Provide clear written feedback on agent performance and usability.

Skills

Attention to detail

Written communication

Documentation

Education

Bachelor's degree

Tools

Screen recording

Job description

Role Title: AI Agent Evaluator

Role Type: Contractor
Location: United States

We are seeking detail-oriented professionals to evaluate AI agents performing everyday tasks. You'll test two AI agents side by side, assess their performance, and provide feedback to improve their reliability and user experience.

Key Responsibilities

• Test AI agents on everyday tasks, including shopping, travel, bookings, bill payments, email follow-ups, and forms.
• Approve or stop agents at key steps.
• Screen-record each testing session.
• Score task completion, quality, control issues, user satisfaction, and time.
• Provide clear written feedback on agent performance and usability.

Required Qualifications

• Based in the United States with a bachelor's degree.
• Regularly use AI tools in daily life.
• 2–3 years of experience, preferably in engineering, business management, or office operations.
• Strong attention to detail and ability to follow instructions.
• Excellent written communication and documentation skills.
• An active personal AI account (Free, Go, or Plus; not Pro) and a smartphone capable of screen recording.

Preferred: Relevant domain expertise. No coding or AI development experience is required.

Project Details

• Task duration: Approximately 1.5 hours per run, including evidence and scoring.
• Kickoff: Monday, October 12.
• Onboarding: Before the project begins, subject to scheduling.