Remote Freelance: AI Agent Evaluation Specialist

Mindrift; United Kingdom; Remote; GBP 19,000 - 31,000; Part time

Benefits offered by this job

Fully remote freelance role

Job summary

Mindrift is seeking a remote freelance evaluator specialist to design realistic test scenarios for LLM-based agents. You will craft structured tasks and gold-standard behaviors that guide assessments and comparisons across projects.

You'll annotate steps, expected outputs, and edge cases, then work with developers to test and refine scenarios. A strong analytical mindset and attention to detail are essential.

Qualifications

• Bachelor's or Master's degree in CS/SE/DS/AI/NLP or related field.
• Strong English (C1+).
• Able to learn new methods and handle complex guidelines.

Responsibilities

• Design structured evaluation scenarios for LLM-based agents.
• Define golden path and acceptable agent behavior.
• Annotate task steps, outputs, and edge cases.
• Collaborate with developers to test and improve clarity.
• Review agent outputs and adapt tests accordingly.

Skills

Test scenario design

LLM evaluation

Analytical thinking

Education

Bachelor's/Master's in CS/AI/NLP

Job description

Mindrift is seeking a remote freelance evaluator specialist to design realistic test scenarios for LLM-based agents. You will craft structured tasks and gold-standard behaviors that guide assessments and comparisons across projects.

You'll annotate steps, expected outputs, and edge cases, then work with developers to test and refine scenarios. A strong analytical mindset and attention to detail are essential.