Remote AI Evaluation Scenario Designer

Mindrift; Canada; Remote; CAD 61,992; Part time

Benefits offered by this job

Flexible work schedule

Competitive pay up to $45/hour

Opportunity to enhance your portfolio

Influence future AI model development

Job summary

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents. The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experience with QA and data analysis, and strong communication skills in English. This is a flexible freelance position that allows working remotely, with rates up to $45/hour based on skills and experience. Engage in cutting-edge AI projects and enhance your professional portfolio.

Qualifications

• Background in QA, software testing, data analysis, or NLP annotation.
• Comfortable with structured formats like JSON/YAML for scenario description.
• Curious and open to working with AI-generated content and agent logs.

Responsibilities

• Create structured test cases that simulate complex human workflows.
• Define gold-standard behavior and scoring logic to evaluate agent actions.
• Analyze agent logs, failure modes, and decision paths.

Skills

Analytical mindset

Attention to detail

Good understanding of test design principles

Strong written communication skills in English

Basic experience with Python

Basic experience with JS

Education

Bachelor's and/or Master's Degree in Computer Science, Software Engineering, Data Science, AI, or related fields

Tools

JSON

YAML

Job description

A leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents. The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experience with QA and data analysis, and strong communication skills in English. This is a flexible freelance position that allows working remotely, with rates up to $45/hour based on skills and experience. Engage in cutting-edge AI projects and enhance your professional portfolio.