Remote Freelance: AI Agent Evaluation Designer

Mindrift (Toloka); United States; Remote; USD 25,000 - 41,000; Part time

Job summary

Mindrift (Toloka) is seeking a freelancer to design realistic evaluation scenarios for LLM-based agents. You will craft test cases, define gold-standard behaviour, annotate steps, and ensure each scenario is clear, scored, and reusable.

The role is part-time and fully remote, with rates up to $30/hour depending on skills and project needs. Ideal candidates have a STEM degree, strong English, and an interest in how AI agents make decisions.

Qualifications

• Bachelor's or Master's in CS/AI or related field.
• Advanced English communication and documentation skills.
• Interest in how AI agents make decisions.

Responsibilities

• Design structured test scenarios based on real-world tasks.
• Define the golden path and acceptable agent behaviour.
• Annotate task steps, expected outputs, and edge cases.
• Collaborate with developers to test scenarios and improve clarity.
• Review agent outputs and adapt tests accordingly.

Skills

Test scenario design

Gold-standard definition

Edge case annotation

Collaboration with developers

Agent evaluation

Education

Bachelor's or Master's degree in CS/AI

Job description

Mindrift (Toloka) is seeking a freelancer to design realistic evaluation scenarios for LLM-based agents. You will craft test cases, define gold-standard behaviour, annotate steps, and ensure each scenario is clear, scored, and reusable.

The role is part-time and fully remote, with rates up to $30/hour depending on skills and project needs. Ideal candidates have a STEM degree, strong English, and an interest in how AI agents make decisions.