Senior QA Engineer: AI Testing & Agentic Systems

Jobtailor; Austin (TX); On-site; USD 120,000 - 150,000; Full time

Job summary

Jobtailor is seeking a senior QA leader to own the quality strategy for teams shipping largely agent-generated code. You will shape testing priorities for AI-powered features and define how we measure success across rapid development cycles.

Responsibilities include building automated test infrastructures across unit, integration, end-to-end, and performance tests; defining evaluation frameworks; and partnering with engineering to ship reliable, safe AI-enabled products with robust test data and

Qualifications

• 8+ years of industry experience in test automation or quality engineering.
• Strong programming skills in Python, TypeScript, Java, or C#.
• Hands-on experience testing AI and model-based systems, including evaluation frameworks.
• Ability to build and scale automated testing platforms, including CI/CD integration.
• Excellent communication and collaboration across distributed teams.
• Experience with API testing, performance testing, and AI failure modes.

Responsibilities

• Own the quality strategy for teams shipping largely agent-generated code.
• Build and maintain automated test infrastructure across unit, integration, end-to-end, contract, API, performance, and regression testing.
• Define and operate evaluation frameworks for AI-powered features, including datasets, scoring systems, model-as-judge approaches, human calibration, regression tracking, and release gates.
• Design testing strategies for non-deterministic and agentic systems.
• Use AI coding agents to generate and maintain meaningful, risk-based test suites.
• Test agent behavior, including multi-step task execution, tool usage, failure recovery, safety enforcement, and human handoff workflows.
• Instrument production quality signals and use real-world feedback to improve evaluation and testing frameworks.
• Partner with engineering teams throughout the development lifecycle, from refinement through release.

Skills

Test automation

Python programming

TypeScript programming

Java programming

C# programming

AI testing

CI/CD integration

Flaky test management

Agentic systems evaluation

API testing

Performance testing

Communication across distributed teams

Tools

MCP Tooling

Evaluation Frameworks

Test Data Strategies

Agentic Development Environments

API Testing Tools

Job description

Jobtailor is seeking a senior QA leader to own the quality strategy for teams shipping largely agent-generated code. You will shape testing priorities for AI-powered features and define how we measure success across rapid development cycles.

Responsibilities include building automated test infrastructures across unit, integration, end-to-end, and performance tests; defining evaluation frameworks; and partnering with engineering to ship reliable, safe AI-enabled products with robust test data and