AI Agent Evaluation Specialist

OpenTrain AI, Inc.; Northern (KY); Remote; USD 34,000 - 62,000; Part time

Job summary

OpenTrain AI, Inc. is seeking a remote, part-time contractor to test two AI agents on a smartphone, completing realistic errands and recording outcomes. You will evaluate task completion, quality, control issues, user satisfaction, and time, then provide clear feedback.

The role requires a Bachelor's degree, strong written English, 2–3 years of relevant experience, and habitual use of AI tools. Flexible 1.5-hour runs with detailed documentation are expected.

Qualifications

• Bachelor's degree required.
• Two to three years of relevant experience.
• Regular use of AI tools in daily life.
• Strong attention to detail and reliable instruction-following.
• Clear written communication and organized documentation.
• Ability to record a smartphone screen.
• Active personal AI accounts that can be used on a phone.
• Generalists welcome; experience in related areas is helpful.
• No coding background or AI industry experience required.

Responsibilities

• Test two AI agents side by side on a smartphone.
• Record and score runs for completion, quality, control, satisfaction, and time.
• Write clear notes on what worked and what failed.
• Follow detailed instructions and document every step accurately.
• Approve or stop an agent when critical actions are needed.

Skills

Strong written English communication

Attention to detail

Instruction-following

Documentation skills

Education

Bachelor's degree

Tools

Screen recording on smartphone

Job description

The Work

You will test two AI agents side by side on a smartphone while completing realistic everyday errands. Your feedback will show where agents complete tasks well, lose control, or create a poor user experience.

• Complete tasks such as paying bills, processing returns, sending follow-up emails, making bookings, shopping, managing groceries, planning travel, coordinating family logistics, attending outings, and completing forms.
• Use personal accounts to run realistic scenarios with both AI agents.
• Approve or stop an agent when an important action requires your control.
• Screen record every attempt from a smartphone.
• Score each run for completion, quality, control issues, satisfaction, and time.
• Write clear notes about what worked, what failed, and how the experience felt.
• Follow detailed instructions and document every step accurately.

What It Pays and Takes

This is a remote, part-time contractor role completed from a smartphone. Each run takes about 1.5 hours, including evidence collection and scoring, and the schedule is flexible.

• Pay: $25 to $45 per hour.
• Location: Open to candidates in the United States.
• Language: Strong written English communication.
• Education: Bachelor's degree required.
• Experience: Two to three years of relevant experience.
• Regular use of AI tools in daily life.
• Strong attention to detail and reliable instruction-following.
• Clear written communication and organized documentation.
• Ability to record a smartphone screen.
• Active personal AI accounts that can be used on a phone.
• Generalists are welcome. Experience in engineering, business management, office operations, digital workflows, or another area related to everyday errands is helpful.
• No coding background or previous AI industry experience is required.

About AI Training Work

AI training is the human work behind artificial intelligence, including testing systems, rating their responses, and recording clear feedback. People who carefully evaluate real-world behavior help make AI tools more accurate, useful, and easier to control.