Evaluation Scenario Writer - QA

Mindrift; New York (NY); Remote; USD 62,508 - 89,027; Part time

Benefits offered by this job

Competitive pay up to $55/hour

Flexible remote work

Experience with advanced AI projects

Job summary

A leading technology firm is seeking an Evaluation Scenario Writer - QA to ensure the quality of evaluation scenarios for AI projects. This part-time, flexible role involves reviewing tests, spotting inconsistencies, and collaborating on automation. Ideal for those with a robust QA background, the position offers competitive rates up to $55/hour and valuable experience in advanced AI environments.

Qualifications

• Strong QA background (manual or automation), preferably in complex testing environments.
• Ability to evaluate logic and structure of test scenarios (even if written by others).
• Clear communication and documentation skills.

Responsibilities

• Reviewing and validating test scenarios from Evaluation Writers.
• Spotting logical inconsistencies, ambiguities, or missing checks.
• Collaborating with infrastructure and tool developers to automate parts of the review.

Skills

QA background (manual or automation)

Understanding of test design

Ability to evaluate logic and structure

Experience reviewing structured test case formats

Familiarity with Python and JS scripting

Clear communication skills

Willingness to write or refactor test scenarios

Experience testing AI-based systems

Git/GitHub workflow familiarity

Experience using test management tools

Job description

Overview

At Mindrift, we connect domain experts with cutting-edge AI projects. We are seeking an Evaluation Scenario Writer - QA for a project focused on ensuring the quality and correctness of evaluation scenarios created for LLM agents. This is a flexible, project-based opportunity that blends manual scenario validation, automated test thinking, and collaboration with writers and engineers.

Responsibilities

• Reviewing and validating test scenarios from Evaluation Writers
• Spotting logical inconsistencies, ambiguities, or missing checks
• Suggesting improvements to structure, edge cases, or scoring logic
• Collaborating with infrastructure and tool developers to automate parts of the review
• Creating clean and testable examples for others to follow

Qualifications

• Strong QA background (manual or automation), preferably in complex testing environments
• Understanding of test design, regression testing, and edge case detection
• Ability to evaluate logic and structure of test scenarios (even if written by others)
• Experience reviewing and debugging structured test case formats (JSON, YAML)
• Familiarity with Python and JS scripting for test automation or validation
• Clear communication and documentation skills
• Willingness to occasionally write or refactor test scenarios
• Experience testing AI-based systems or NLP applications (valued)
• Familiarity with scoring systems and behavioral evaluation
• Git/GitHub workflow familiarity (PR review, versioning of test cases)
• Experience using test management systems or tracking tools

Benefits

• Get paid for your expertise, with rates that can go up to $55/hour depending on your skills, experience, and project needs
• Take part in a flexible, remote, freelance project that fits around your commitments
• Participate in an advanced AI project and gain valuable experience for your portfolio
• Influence how future AI models understand and communicate in your field of expertise

How To Get Started

Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule.

Employment details

• Seniority level: Internship
• Employment type: Part-time