Test Engineer

Airo Digital Labs, LLC; United States; On-site; USD 90,000 - 120,000; Full time

Job summary

Airo Digital Labs, LLC is seeking an SDET/QA Automation Engineer to own end-to-end test automation across Python and JVM services and React UIs within our Kentucky-based team. You will build and maintain automated test suites, focusing on API/contract testing and CI/CD integration.

The role includes an AI-eval focus for evaluation harnesses and data-quality testing for pipelines, with an emphasis on reliability and automation quality.

Qualifications

• 3–6 years as an SDET/automation-focused QA engineer.
• Shipped automation running in CI for microservices; comfortable reading application code.

Responsibilities

• Build and maintain automated test suites across the Python and JVM services and React UIs.
• Own API, contract and integration testing for the Orchestrator, Workflow Builder, Connectors and MCP / A2A surfaces.
• Integrate tests into CI/CD gates; report coverage and quality signals.
• AI-eval focus: design and run evaluation harnesses for the Signal / Action Engines and Librarian Agent — accuracy, hallucination, regression and guardrail checks for LLM/agent outputs.
• Performance & data focus: build load and resilience tests for orchestration and event driven flows, and data-quality tests for pipelines.

Skills

Pytest

Playwright

Selenium

REST Assured

JUnit

Postman

Pact

CI/CD

LLM evaluation

K6

Tools

REST Assured

JUnit

Job description

Own quality across the POD through automation. Both cover functional, API and integration testing.

Key Responsibilities

• Build and maintain automated test suites across the Python and JVM services and React UIs.
• Own API, contract and integration testing for the Orchestrator, Workflow Builder, Connectors and MCP / A2A surfaces.
• Integrate tests into CI/CD gates; report coverage and quality signals.
• AI-eval focus: design and run evaluation harnesses for the Signal / Action Engines and Librarian Agent — accuracy, hallucination, regression and guardrail checks for LLM/agent outputs.
• Performance & data focus: build load and resilience tests for orchestration and event driven flows, and data-quality tests for pipelines.

Must-have Skills & Technologies

• Test automation with Pytest / Playwright / Selenium and a JVM framework (REST Assured, JUnit) for API and UI testing.
• API and contract testing (Postman, Pact) and CI integration.
• One of: LLM/agent evaluation frameworks and techniques (for the AI-eval role) or performance testing (k6, JMeter, Locust) and data-quality validation (for the performance/data role).

Prior Experience

• 3–6 years as an SDET / automation-focused QA engineer.
• Shipped automation running in CI for microservices; comfortable reading application code.

Nice to Have

• Experience testing event-driven / workflow systems (Temporal, Kafka / Event Hub).
• Familiarity with AI/LLM quality practices or data observability.