AI QA Engineer: Release Readiness & Quality Monitoring

Ai Makers; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

You will build evaluation sets, design regression tests across workflows, review edge cases, verify outcomes, and surface signals of drift and rising token costs. Clear reports will support release decisions and stakeholder understanding.

Qualifications

• Ability to test AI systems whose outputs vary using realistic evaluation sets.
• Design regression tests across connected workflows and identify silent downstream failures.
• Judgment to assess edge cases and determine if outputs are fit for business use.
• Ability to measure and verify claimed outcomes with a repeatable method.
• Clear written communication for documenting findings and release decisions.

Responsibilities

• Build and maintain evaluation sets for AI systems, updating them as real user traffic reveals new questions and edge cases.
• Create regression checks for workflow changes, including downstream failures caused by altered outputs or formats.
• Review business scenarios and client edge cases, assess whether results are usable, and make the release readiness decision.
• Design a repeatable method to verify reported business outcomes on individual projects and flag claims that do not hold.
• Monitor for quality drift and rising token costs, and create signals that surface changes early.
• Write concise acceptance reports, defect notes, and drift findings that stakeholders can understand quickly.

Skills

Test AI systems

Regression testing design

Edge-case judgment

Measurement & verification

Communication skills

Job description

You will build evaluation sets, design regression tests across workflows, review edge cases, verify outcomes, and surface signals of drift and rising token costs. Clear reports will support release decisions and stakeholder understanding.