AI Agent Testing & Evaluation Scenario Architect

Mindrift; Houston (TX); Remote; USD 90,921 - 129,494; Part time

Benefits offered by this job

Get paid for your expertise

Flexible work schedule

Opportunity to work on advanced AI projects

Job summary

A leading AI consultancy is seeking a detail-oriented individual to design evaluation scenarios for LLM-based agents. This part-time, remote role allows you to create structured test cases while working flexibly around your commitments. The ideal candidate holds a relevant degree and has experience in QA or data analysis, with a strong command of English. Compensation can go up to $80/hour depending on skills and project needs, offering an excellent opportunity to contribute to advanced AI projects.

Qualifications

• Bachelor's or Master's in a relevant field.
• Background in QA, software testing, or data analysis.
• Good understanding of test design principles.
• Strong written communication skills in English.
• Familiarity with structured formats like JSON/YAML.
• Can define expected agent behaviors.
• Basic experience with Python and JavaScript.

Responsibilities

• Create structured test cases that simulate complex human workflows.
• Define gold‑standard behavior and scoring logic.
• Analyze agent logs and decision paths.
• Work with test frameworks to validate scenarios.
• Ensure scenarios are production‑ready and reusable.

Skills

Analytical mindset

Attention to detail

Written communication

Experience with JSON/YAML

Basic experience in Python

Basic experience in JavaScript

Curiosity about AI

Education

Bachelor's or Master's Degree in Computer Science, Software Engineering, Data Science, or related fields

Job description

A leading AI consultancy is seeking a detail-oriented individual to design evaluation scenarios for LLM-based agents. This part-time, remote role allows you to create structured test cases while working flexibly around your commitments. The ideal candidate holds a relevant degree and has experience in QA or data analysis, with a strong command of English. Compensation can go up to $80/hour depending on skills and project needs, offering an excellent opportunity to contribute to advanced AI projects.