Lead Quality Automation Engineer – AI Platform

InSite; Washington (District of Columbia); On-site; USD 120,000 - 170,000; Full time

Job summary

InSite seeks a Senior Test Automation Engineer & Quality Engineering Lead to establish and scale quality engineering for a modern AI platform and AI-enabled applications. This hands-on, player-coach role defines automation strategy, builds reusable frameworks, and integrates quality into CI/CD across web, mobile, APIs, data, and AI workflows.

The role covers AI agent evaluation, data quality validation, and cross-team collaboration with software, platform, data, AI, and DevOps engineers.

Qualifications

• Senior-level test automation or quality engineering experience, including designing automation frameworks from the ground up.
• Strong Python skills for API, service, data, and AI test automation.
• Strong web, API/service, integration, and end-to-end automation experience.
• Experience with automated testing of iOS and Android applications; cross-platform mobile automation is desirable.

Responsibilities

• Define an automation-first strategy across UI, APIs, services, integrations, data, and AI workflows.
• Build reusable automation frameworks and high-value regression suites, with API/service-level testing as the foundation.
• Automate critical web, mobile, and end-to-end journeys, including authentication, authorization, multi-tenant behavior, asynchronous interactions, and failure scenarios.
• Integrate automated tests and quality gates into CI/CD pipelines.
• Establish test execution monitoring, failure triage, reporting, and flaky-test management.

Skills

Python

API testing

Web automation

Mobile automation

SQL

AWS

OpenAI APIs

Generative AI

CI/CD automation

Debugging

Tools

Docker

Bitbucket Pipelines

Job description

We are seeking a Senior Test Automation Engineer & Quality Engineering Lead to establish and scale quality engineering for a modern AI platform and AI-enabled applications.

This is a hands-on, player-coach role responsible for defining the automation strategy, building reusable frameworks and critical test suites, integrating quality into CI/CD, and helping engineering teams increasingly own product quality.

The role spans web and mobile applications, APIs and backend services, data and integrations, and AI/agent behavior.

Key Responsibilities

• Define an automation-first strategy across UI, APIs, services, integrations, data, and AI workflows.
• Build reusable automation frameworks and high-value regression suites, with API/service-level testing as the foundation.
• Automate critical web, mobile, and end-to-end journeys, including authentication, authorization, multi-tenant behavior, asynchronous interactions, and failure scenarios.
• Integrate automated tests and quality gates into CI/CD pipelines.
• Establish test execution monitoring, failure triage, reporting, and flaky-test management.
• Automate critical performance, latency, resilience, concurrency, and dependency-failure scenarios.

AI, Agent & Data Quality

• Establish repeatable evaluation and regression testing for Generative AI and agentic applications.
• Validate appropriate tool and data usage, workflow and authorization boundaries, human-approval controls, and handling of missing or conflicting information.
• Validate groundedness, factual accuracy, relevance, evidence quality, hallucination risk, instruction adherence, latency, and reliability.
• Test retrieval, tool-calling, multi-step agent workflows, and AI application failure scenarios.
• Automate data-quality validation covering freshness, integrity, completeness, transformation, traceability, and tenant isolation.
• Establish representative synthetic and simulated datasets for deterministic regression and AI evaluation.

Engineering Leadership

• Build the initial automation architecture and highest-value test suites.
• Lead automation design, code quality, and engineering standards.
• Partner with software, platform, data, AI, and DevOps engineers to embed testability early.
• Participate in architecture and design reviews and mentor engineers contributing to shared automation frameworks.
• Promote the right mix of unit, API, integration, end-to-end, performance, exploratory, and AI evaluation testing while driving distributed quality ownership.

Experience & Technology Skills

Strong candidates will have:

• Senior-level test automation or quality engineering experience, including designing automation frameworks from the ground up.
• Strong Python skills for API, service, data, and AI test automation.
• Strong web, API/service, integration, and end-to-end automation experience.
• Experience with automated testing of iOS and Android applications; cross-platform mobile automation is desirable.
• Strong experience testing REST APIs, asynchronous services, authentication, authorization, and multi-tenant systems.
• Strong SQL and data-validation skills across relational, operational, time-series, and document-oriented data.
• Hands-on experience testing applications and services deployed on AWS; AWS application, data, container, and observability services are strongly preferred.
• Experience testing applications built with OpenAI models/APIs or comparable production LLM platforms; OpenAI experience preferred.
• Hands-on experience with Generative AI and agentic systems, including retrieval, grounding, tool-calling, multi-step workflows, behavioral regression, hallucination/evidence validation, and AI observability.
• Experience integrating automated testing into Git-based CI/CD workflows; Bitbucket and Bitbucket Pipelines experience preferred.
• Experience with Docker/containerized applications and container-based test environments preferred.
• Strong debugging and failure-analysis skills using application logs, metrics, traces, APIs, data, and infrastructure.
• Candidates should be comfortable contributing production-quality automation code and working directly with software engineers.

Success in the Role

• Reusable automation frameworks and quality standards.
• Strong API/service regression coverage with focused UI and end-to-end automation.
• Continuous data-quality and tenant-isolation validation.
• Repeatable AI/agent evaluation and behavioral regression testing.
• Actionable CI/CD quality signals and rapid failure diagnosis.
• Increasing quality ownership across engineering teams.

The goal is to provide fast, trustworthy quality signals that enable teams to release AI-enabled products confidently and efficiently.