Remote Freelance AI Agent Evaluator — Craft Test Scenarios

Mindrift (Toloka); United States; Híbrido; USD 3.444.000 - 4.133.000; A tiempo parcial

Ventajas ofrecidas por este puesto de trabajo

Remote freelance

Descripción de la vacante

Mindrift is seeking a detail-oriented, analytical freelancer to design structured evaluation scenarios for LLM-based agents. You will create test cases, define gold-standard behavior, and ensure scenarios are clear, scored, and reusable.

The role requires a strong analytical mindset, meticulousness, and the ability to learn new methods. The position is fully remote, freelance, with rates up to $30/hour depending on skills and project needs.

Formación

• Require a Bachelor’s or Master’s degree in CS/SE/AI/NLP or related field.
• Advanced English (C1 or above) is required.
• Freelance remote role; you will need a laptop and stable internet connection.

Responsabilidades

• Designing structured test scenarios based on real-world tasks
• Defining the golden path and acceptable agent behavior
• Annotating task steps, expected outputs, and edge cases
• Working with developers to test your scenarios and improve clarity
• Reviewing agent outputs and adapting tests accordingly

Conocimientos

Test design

Attention to detail

Analytical mindset

Strong communication

Adaptability

Educación

Bachelor's or Master’s in CS/SE/AI/NLP

Descripción del empleo

Mindrift is seeking a detail-oriented, analytical freelancer to design structured evaluation scenarios for LLM-based agents. You will create test cases, define gold-standard behavior, and ensure scenarios are clear, scored, and reusable.

The role requires a strong analytical mindset, meticulousness, and the ability to learn new methods. The position is fully remote, freelance, with rates up to $30/hour depending on skills and project needs.