Remote AI Agent Evaluation Specialist

Remote Worker LTD.; United States; On-site; USD 28,000 - 41,000; Part time

Benefits offered by this job

Remote freelance project

Job summary

Mindrift is seeking a detail‑driven designer of evaluation scenarios for LLM agents. This fully remote freelance role focuses on creating structured test cases, scoring rubrics, and gold-standard behaviors to benchmark AI performance.

You’ll collaborate with developers, annotate steps and edge cases, and adapt tests as needed. Requires a CS/AI-related degree and advanced English (C1+); remote work with flexible hours for part-time projects.

Qualifications

• Bachelor's or Master's degree in Computer Science, Software Engineering, Data Science / Data Analytics, AI / ML, NLP, IS or related fields.
• Advanced English (C1) or higher.
• Willing to learn new methods and switch between tasks quickly with complex guidelines.
• Freelance role is fully remote; requires laptop and internet connection.

Responsibilities

• Design structured test scenarios based on real-world tasks.
• Define the golden path and acceptable agent behavior.
• Annotate task steps, expected outputs, and edge cases.
• Collaborate with devs to test scenarios and improve clarity.
• Review agent outputs and adapt tests accordingly.

Skills

Analytical mindset

Detail oriented

Remote work

Education

Bachelor's or Master’s in CS/SE/Data Science/AI/NLP

Job description

Mindrift is seeking a detail‑driven designer of evaluation scenarios for LLM agents. This fully remote freelance role focuses on creating structured test cases, scoring rubrics, and gold-standard behaviors to benchmark AI performance.

You’ll collaborate with developers, annotate steps and edge cases, and adapt tests as needed. Requires a CS/AI-related degree and advanced English (C1+); remote work with flexible hours for part-time projects.