AI Agent Evaluator, Real-World Task Testing (Remote)

Vettedlance; United States; Remote; USD 28,000 - 55,000; Part time

Job summary

Vettedlance is seeking detail-oriented contributors to craft realistic AI task scenarios and evaluate agent performance. You will describe steps, judge outcomes, and document failures in plain language to help developers improve automation.

The role is remote, part-time, hourly, 10–40 hours per week, paid weekly in USD. A short paid assessment will determine your rate within $20–$40 per hour.

Qualifications

• Strong analytical and logical-reasoning skills.
• Clear written English and careful attention to detail.
• Comfort with everyday web apps, spreadsheets and productivity tools.
• Able to follow detailed guidelines and work independently.

Responsibilities

• Write realistic task scenarios for AI agents performing browser, app, email and calendar tasks.
• Watch agent runs and record whether each step and the final result are correct.
• Rate outcomes against a rubric and explain failures in plain language.
• Research claims and verify data the agent used or produced.

Skills

Analytical thinking

Written English

Attention to detail

Independent worker

Follow detailed guidelines

Tools

Web apps

Spreadsheets

Productivity tools

Job description

Put AI agents through realistic multi-step jobs and judge whether they actually finish them. Your scenarios, ratings and notes show developers where an agent clicks wrong, skips a step or reports success it did not earn.

What you will do

• Write realistic task scenarios for agents that browse, use apps or handle email and calendars


• Watch agent runs and record whether each step and the final result are correct


• Rate outcomes against a rubric and explain every failure in plain language


• Research claims and check data the agent used or produced

What you need

• Strong analytical and logical-reasoning skills


• Clear written English and careful attention to detail


• Comfort with everyday web apps, spreadsheets and productivity tools


• Able to follow detailed guidelines and work independently

Contract

• Remote, hourly contract (independent contractor)


• Part-time, flexible: 10-40 hrs/week


• Remote

Pay: $20 - $40 / hour (USD). Paid weekly in USD. Typical rate $20-$40 per hour, confirmed after a short paid assessment.