QA Engineer

Ai Makers; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

You will build evaluation sets, design regression tests across workflows, review edge cases, verify outcomes, and surface signals of drift and rising token costs. Clear reports will support release decisions and stakeholder understanding.

Qualifications

• Ability to test AI systems whose outputs vary using realistic evaluation sets.
• Design regression tests across connected workflows and identify silent downstream failures.
• Judgment to assess edge cases and determine if outputs are fit for business use.
• Ability to measure and verify claimed outcomes with a repeatable method.
• Clear written communication for documenting findings and release decisions.

Responsibilities

• Build and maintain evaluation sets for AI systems, updating them as real user traffic reveals new questions and edge cases.
• Create regression checks for workflow changes, including downstream failures caused by altered outputs or formats.
• Review business scenarios and client edge cases, assess whether results are usable, and make the release readiness decision.
• Design a repeatable method to verify reported business outcomes on individual projects and flag claims that do not hold.
• Monitor for quality drift and rising token costs, and create signals that surface changes early.
• Write concise acceptance reports, defect notes, and drift findings that stakeholders can understand quickly.

Skills

Test AI systems

Regression testing design

Edge-case judgment

Measurement & verification

Communication skills

Job description

Role overview

Establish quality controls for AI systems running in production. This role focuses on evaluating variable model outputs and connected workflows, keeping tests relevant to real usage, and determining whether a system is ready for delivery. It also tracks reliability, business claims, and gradual changes in quality or cost.

Responsibilities

• Build and maintain evaluation sets for AI systems, updating them as real user traffic reveals new questions and edge cases.
• Create regression checks for workflow changes, including downstream failures caused by altered outputs or formats.
• Review business scenarios and client edge cases, assess whether results are usable, and make the release readiness decision.
• Design a repeatable method to verify reported business outcomes on individual projects and flag claims that do not hold.
• Monitor for quality drift and rising token costs, and create signals that surface changes early.
• Write concise acceptance reports, defect notes, and drift findings that stakeholders can understand quickly.

Requirements

• Ability to test AI systems whose outputs vary, using evaluation sets that reflect realistic user behavior.
• Skill in designing regression tests across connected workflows and identifying silent downstream failures.
• Judgment to assess edge cases and determine whether outputs are fit for business use.
• Ability to measure and verify claimed outcomes with a repeatable, project-level method.
• Attention to operational signals such as declining quality and increasing model costs.
• Clear written communication for documenting findings, release decisions, and non-conformities.