Automation Test Engineer - API, AI Eval & CI

Airo Digital Labs, LLC; Northern (KY); On-site; USD 90,000 - 120,000; Full time

Job summary

Airo Digital Labs, LLC is seeking an SDET/QA Automation Engineer to own end-to-end test automation across Python and JVM services and React UIs within our Kentucky-based team. You will build and maintain automated test suites, focusing on API/contract testing and CI/CD integration.

The role includes an AI-eval focus for evaluation harnesses and data-quality testing for pipelines, with an emphasis on reliability and automation quality.

Qualifications

• 3–6 years as an SDET/automation-focused QA engineer.
• Shipped automation running in CI for microservices; comfortable reading application code.

Responsibilities

• Build and maintain automated test suites across the Python and JVM services and React UIs.
• Own API, contract and integration testing for the Orchestrator, Workflow Builder, Connectors and MCP / A2A surfaces.
• Integrate tests into CI/CD gates; report coverage and quality signals.
• AI-eval focus: design and run evaluation harnesses for the Signal / Action Engines and Librarian Agent — accuracy, hallucination, regression and guardrail checks for LLM/agent outputs.
• Performance & data focus: build load and resilience tests for orchestration and event driven flows, and data-quality tests for pipelines.

Skills

Pytest

Playwright

Selenium

REST Assured

JUnit

Postman

Pact

CI/CD

LLM evaluation

K6

Tools

REST Assured

JUnit

Job description

Airo Digital Labs, LLC is seeking an SDET/QA Automation Engineer to own end-to-end test automation across Python and JVM services and React UIs within our Kentucky-based team. You will build and maintain automated test suites, focusing on API/contract testing and CI/CD integration.

The role includes an AI-eval focus for evaluation harnesses and data-quality testing for pipelines, with an emphasis on reliability and automation quality.