Remote LLM QA Scenario Designer for AI Agent Testing

Mindrift; Remote; ZAR 200 000 - 300 000; Part time

Job summary

An innovative tech company seeks a candidate to design evaluation scenarios for LLM-based agents. The role involves creating structured test cases that mimic human tasks and defining expected behaviors for agent actions. Requires a strong analytical mindset, attention to detail, and comfort in structured formats. This fully remote freelance role offers flexible hours, with payment rates up to $22/hour based on skills and experience.

Benefits

Flexible work schedule

Competitive pay

Remote work opportunity

Experience enhancing portfolio

Qualifications

• Bachelor's and/or Master's Degree in Computer Science, Software Engineering or related fields.
• Background in QA, software testing, data analysis, or NLP annotation.
• Good understanding of test design principles.
• Comfortable with structured formats like JSON/YAML.

Responsibilities

• Design structured test scenarios based on real-world tasks.
• Define the golden path and acceptable agent behavior.
• Annotate task steps, expected outputs, and edge cases.
• Review agent outputs and adapt tests accordingly.

Skills

Analytical mindset

Attention to detail

Strong written communication skills in English

Curiosity about AI

Education

Bachelor's and/or Master's Degree in relevant fields

Tools

Python

JavaScript

JSON

YAML

Job description

An innovative tech company seeks a candidate to design evaluation scenarios for LLM-based agents. The role involves creating structured test cases that mimic human tasks and defining expected behaviors for agent actions. Requires a strong analytical mindset, attention to detail, and comfort in structured formats. This fully remote freelance role offers flexible hours, with payment rates up to $22/hour based on skills and experience.