Senior/Staff QA Engineer – Agentic AI Platform

Ontrac Solutions; Chicago (IL); On-site; USD 120,000 - 150,000; Full time

Job summary

Ontrac Solutions is seeking a Senior/Staff QA Engineer to own end-to-end quality for a critical AI product. You will write precise manual tests and build automated coverage, collaborating with Delivery Lead/Solutions Architect and customer engineering teams.

Ideal candidates have 5+ years of QA, with 2+ at senior level, strong ADLC understanding, UI/UX sensibility, and hands-on Playwright-based automation across CI/CD pipelines, Docker, and modern defect tracking.

Qualifications

• 5+ years of professional QA experience, with at least 2+ years at a senior or staff level.
• Demonstrated experience testing production software through a full ADLC.
• Experience testing AI Agent use cases – conversational flows, tool calling, edge-case behavior.
• Proven experience with automated testing frameworks and building maintainable test suites.
• Ability to stand up and manage a Playwright cluster for E2E testing against a live agent platform.
• Autonomy to define clear test cases from evolving requirements.
• Experience driving full E2E QA lifecycle ownership: test planning, execution, defect triage, regression, sign-off.

Responsibilities

• Own end-to-end QA lifecycle, from planning to sign-off.
• Develop and execute test plans for AI agent workflows.
• Build and maintain automated coverage with Playwright in CI/CD environments.

Skills

Playwright

JavaScript/TypeScript

Python

AI/LLM Testing

UI/UX

Autonomy

Lifecycle Ownership

Defect Tracking

Tools

Git

Jira

Docker

GitHub Actions

Job description

Overview

We are seeking a highly skilled and experienced Senior/Staff QA Engineer to support a critical customer engagement. This role is ideal for a detail-oriented tester who thrives owning quality end-to-end — someone equally comfortable writing precise manual test cases and building automated coverage for a fast-moving agentic AI product.

The successful candidate will work closely with the Delivery Lead/Solutions Architect and customer engineering stakeholders to ensure product quality from initial test planning through final release.

Required Credentials

• 5+ years of professional QA experience, with at least 2+ years at a senior or staff level.
• Demonstrated experience testing production software through a full Application Development Lifecycle (ADLC).

Required Qualifications

• ADLC Expertise: Strong understanding of the Application Development Lifecycle — from requirements through release and support.
• UI/UX Acumen: Solid UI/UX understanding, with the ability to evaluate usability and flag design/interaction issues, not just functional bugs.
• AI Agent Testing: Hands-on experience testing AI Agent use cases — including conversational flows, agent decision-making, tool/function calling, and edge-case or failure-mode behavior.
• Automation Frameworks: Proven experience with automated testing frameworks and building maintainable, scalable test suites.
• Infrastructure Management: Ability to stand up and manage a Playwright cluster to run automated end-to-end (E2E) testing against a live agent platform.
• Autonomy: Ability to independently define clear, well-structured test cases from ambiguous or evolving product requirements.
• Lifecycle Ownership: Experience driving full E2E QA lifecycle ownership: test planning, execution, defect triage, regression, and sign-off.

Required Technical Stack Experience (Suggested Completion)

• Automation Frameworks: Deep expertise with Playwright (including cluster management and parallel execution).
• Languages: Proficiency in JavaScript / TypeScript or Python for writing automation scripts.
• AI/LLM Tools: Familiarity with testing LLM outputs, prompt evaluation frameworks, or conversational AI testing tools.
• CI/CD & DevOps: Experience integrating automated test suites into pipelines (e.g., GitHub Actions, GitLab CI, Jenkins) and working with Docker containerization.
• Defect Tracking & Tools: Proficiency with Git, Jira, and modern test management tools.