AI QA Engineer & Test Automation Lead

Kavaliro; Salt Lake City (UT); On-site; USD 120,000 - 150,000; Full time

Job summary

Kavaliro is seeking a Software Developer in Test for a direct hire role in Salt Lake City, UT. The ideal candidate combines hands-on QA engineering with leadership, able to extend existing automation frameworks and collaborate with developers, PMs, and DevOps.

Responsibilities include building robust automated test suites for API/backend services, validating AI/ML model outputs, integrating tests into GitLab CI/CD, and driving QA metrics across distributed systems, including blockchain

Qualifications

• 10+ years of QA engineering delivery experience with strong focus on backend or API testing.
• 6+ years in Agile SDLC with CI/CD; testing AI/ML or LLM-powered systems, including model validation and regression testing.
• Familiar with AI testing metrics such as precision/recall, drift, bias assessment and online vs offline evaluation.
• Experience with Python, Behave, PyTest, Postman/REST Tools, JMeter or similar tools.
• Solid GitLab CI/CD, Git, Jenkins and Docker experience.

Responsibilities

• Build and implement robust automated test suites for API and backend services.
• Design and implement test strategies for AI/ML models and AI-enabled in-house apps with validation of outputs and determinism.
• Establish automated evaluation frameworks for AI systems and integrate into CI/CD pipelines.
• Partner with AI/ML engineers, platform teams, and governance stakeholders for responsible AI practices and production readiness.
• Integrate automated test suites into existing GitLab CI/CD pipelines and deployment workflows.
• Maintain and improve automation frameworks to support evolving architecture including blockchain components.
• Collaborate with developers, product managers, and DevOps to ensure testability across distributed systems.
• Create and manage defect workflows integrated with JIRA for clear communication and traceability.
• Define, track, and publish QA metrics to support automated test development and quality reporting.
• Build and maintain performance and load testing suites for backend services and smart contract interactions where applicable.
• Perform Functional, API, Automation, Performance, and Integration testing as needed.
• Contribute to continuous improvement of QA processes, tooling, and best practices in Agile.

Skills

QA engineering

AI testing techniques

API testing

Distributed systems testing

Test strategy

Model validation

Prompt testing

Non-deterministic outputs evaluation

Testability & traceability

Tools

Python

Behave

PyTest

Postman

REST tools

JMeter

GitLab CI/CD

Git

Jenkins

Docker

Blockchain testing

Job description

Kavaliro is seeking a Software Developer in Test for a direct hire role in Salt Lake City, UT. The ideal candidate combines hands-on QA engineering with leadership, able to extend existing automation frameworks and collaborate with developers, PMs, and DevOps.

Responsibilities include building robust automated test suites for API/backend services, validating AI/ML model outputs, integrating tests into GitLab CI/CD, and driving QA metrics across distributed systems, including blockchain