Remote QA Evaluation Scenario Architect for LLMs

Mindrift; New York (NY); Remote; USD 62,508 - 89,027; Part time

Benefits offered by this job

Competitive pay up to $55/hour

Flexible remote work

Experience with advanced AI projects

Job summary

A leading technology firm is seeking an Evaluation Scenario Writer - QA to ensure the quality of evaluation scenarios for AI projects. This part-time, flexible role involves reviewing tests, spotting inconsistencies, and collaborating on automation. Ideal for those with a robust QA background, the position offers competitive rates up to $55/hour and valuable experience in advanced AI environments.

Qualifications

• Strong QA background (manual or automation), preferably in complex testing environments.
• Ability to evaluate logic and structure of test scenarios (even if written by others).
• Clear communication and documentation skills.

Responsibilities

• Reviewing and validating test scenarios from Evaluation Writers.
• Spotting logical inconsistencies, ambiguities, or missing checks.
• Collaborating with infrastructure and tool developers to automate parts of the review.

Skills

QA background (manual or automation)

Understanding of test design

Ability to evaluate logic and structure

Experience reviewing structured test case formats

Familiarity with Python and JS scripting

Clear communication skills

Willingness to write or refactor test scenarios

Experience testing AI-based systems

Git/GitHub workflow familiarity

Experience using test management tools

Job description

A leading technology firm is seeking an Evaluation Scenario Writer - QA to ensure the quality of evaluation scenarios for AI projects. This part-time, flexible role involves reviewing tests, spotting inconsistencies, and collaborating on automation. Ideal for those with a robust QA background, the position offers competitive rates up to $55/hour and valuable experience in advanced AI environments.