AI Quality Assurance Engineer — Automation & Reliability

Aquent; Ann Arbor (MI); On-site; USD 110,000 - 150,000; Full time

Benefits offered by this job

Subsidized health plan

Vision plan

Dental plan

401(k) retirement plan with match

Job summary

Aquent Talent is seeking an experienced QA Automation Engineer to lead testing for AI-powered applications, ML systems, AI agents, and generative AI solutions in a financial technology context. You will build scalable test strategies, validate AI outputs for accuracy and safety, and integrate testing into CI/CD pipelines.

You will collaborate with software and AI engineers, product owners, and security teams to ensure reliability, performance, and governance across complex AI workloads, APIs,

Qualifications

• Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related technical field.
• 3–6 years of experience in software testing, QA automation, or quality engineering.
• Proven experience developing automated test solutions using Python, Java, JavaScript/TypeScript, or C#.
• Demonstrated experience with API testing and automation tools.
• Strong understanding of test automation principles, the SDLC, Agile methodologies, and CI/CD pipelines.
• Experience testing distributed systems, web applications, and APIs.
• Proficiency in AI Testing concepts including Prompt Testing, Response Evaluation, Hallucination Detection, Agent Workflow Validation, RAG Validation, AI Safety Testing, AI Regression Testing, and AI Benchmarking.
• Expertise in Quality Engineering practices such as Test Automation, API Testing, Integration Testing, Performance Testing, Load Testing, Security Testing, Defect Analysis, and Root Cause Investigation.
• Familiarity with Selenium, Playwright, Postman, JUnit/PyTest, GitHub Actions, Jenkins, Docker, and Kubernetes.

Responsibilities

• Develop and implement comprehensive testing strategies for AI applications, platforms, and services.
• Validate AI model outputs for accuracy, consistency, reliability, and safety, performing adversarial, negative, and edge‑case testing to identify potential failures and hallucinations.
• Design and execute functional, integration, end‑to‑end, regression, and performance tests specifically tailored for AI solutions.
• Create intricate test cases for prompt‑driven, agentic, and retrieval‑based AI workflows, ensuring robust validation of AI guardrails, business rules, permissions, and governance controls.
• Build and maintain advanced automated test frameworks and evaluation pipelines for AI responses and workflows, integrating AI testing seamlessly into CI/CD pipelines.
• Implement automated quality scoring and regression detection mechanisms, and create reusable test data, mocks, simulators, and validation frameworks.
• Test AI agents, workflows, APIs, complex system integrations, and tool‑calling capabilities, validating integrations with external systems, data sources, and enterprise services.
• Verify performance, reliability, scalability, and resiliency of AI workloads, including executing load and stress testing for AI services.
• Collaborate proactively with software engineers, AI engineers, product owners, architects, and security teams, providing critical quality feedback during design reviews and development.
• Contribute significantly to test strategy, quality standards, and best practices, and support production readiness reviews and defect triage activities.
• Contribute to projects such as testing AI chat assistants and copilots, validating AI agent workflows, evaluating retrieval augmented generation (RAG) search quality, automating AI response evaluation frameworks, and performance testing AI services and orchestration platforms.

Skills

Python

Java

JavaScript/TypeScript

C#

API testing

CI/CD

Selenium

Playwright

Postman

JUnit/PyTest

GitHub Actions

Jenkins

Docker

Kubernetes

Test automation

Education

Bachelor's degree in Computer Science or related field

Tools

Selenium

Playwright

Postman

JUnit/PyTest

GitHub Actions

Jenkins

Docker

Kubernetes

Job description

Aquent Talent is seeking an experienced QA Automation Engineer to lead testing for AI-powered applications, ML systems, AI agents, and generative AI solutions in a financial technology context. You will build scalable test strategies, validate AI outputs for accuracy and safety, and integrate testing into CI/CD pipelines.

You will collaborate with software and AI engineers, product owners, and security teams to ensure reliability, performance, and governance across complex AI workloads, APIs,