Software Developer in Test

Kavaliro; Salt Lake City (UT); On-site; USD 120,000 - 150,000; Full time

Job summary

Kavaliro is seeking a Software Developer in Test for a direct hire role in Salt Lake City, UT. The ideal candidate combines hands-on QA engineering with leadership, able to extend existing automation frameworks and collaborate with developers, PMs, and DevOps.

Responsibilities include building robust automated test suites for API/backend services, validating AI/ML model outputs, integrating tests into GitLab CI/CD, and driving QA metrics across distributed systems, including blockchain

Qualifications

• 10+ years of QA engineering delivery experience with strong focus on backend or API testing.
• 6+ years in Agile SDLC with CI/CD; testing AI/ML or LLM-powered systems, including model validation and regression testing.
• Familiar with AI testing metrics such as precision/recall, drift, bias assessment and online vs offline evaluation.
• Experience with Python, Behave, PyTest, Postman/REST Tools, JMeter or similar tools.
• Solid GitLab CI/CD, Git, Jenkins and Docker experience.

Responsibilities

• Build and implement robust automated test suites for API and backend services.
• Design and implement test strategies for AI/ML models and AI-enabled in-house apps with validation of outputs and determinism.
• Establish automated evaluation frameworks for AI systems and integrate into CI/CD pipelines.
• Partner with AI/ML engineers, platform teams, and governance stakeholders for responsible AI practices and production readiness.
• Integrate automated test suites into existing GitLab CI/CD pipelines and deployment workflows.
• Maintain and improve automation frameworks to support evolving architecture including blockchain components.
• Collaborate with developers, product managers, and DevOps to ensure testability across distributed systems.
• Create and manage defect workflows integrated with JIRA for clear communication and traceability.
• Define, track, and publish QA metrics to support automated test development and quality reporting.
• Build and maintain performance and load testing suites for backend services and smart contract interactions where applicable.
• Perform Functional, API, Automation, Performance, and Integration testing as needed.
• Contribute to continuous improvement of QA processes, tooling, and best practices in Agile.

Skills

QA engineering

AI testing techniques

API testing

Distributed systems testing

Test strategy

Model validation

Prompt testing

Non-deterministic outputs evaluation

Testability & traceability

Tools

Python

Behave

PyTest

Postman

REST tools

JMeter

GitLab CI/CD

Git

Jenkins

Docker

Blockchain testing

Job description

Position Summary

Our client is seeking a Software Developer in Test for a direct hire opportunity. The Software Developer in Test will bring practical knowledge of agile development methodologies and engineering best practices. This role will work closely with the Development and Testing team. The ideal candidate is a combination of a hands‑on test engineer who has some managerial experience, and who can understand existing automation test frameworks and has an ability to enhance and implement the same and should also be able to work with other engineers and product managers to support feature‑based testing.

Pay: $120,000 - $150,000 annually

Responsibilities

• Build and implement robust automated test suites for API and backend services.
• Design and implement test strategies for AI/ML models and AI enabled in-house applications, including validation of model outputs, prompt behavior, determinism, and regression across releases.
• Establish automated evaluation frameworks for AI systems (i.e. accuracy, consistency, hallucination detection, bias, and drift) and integrate them into CI/CD pipelines.
• Partner with AI/ML Engineers, Platform teams, and Governance stakeholders to ensure responsible AI practices, including testability, traceability, and production readiness of AI features.
• Integrate automated test suites into existing GitLab CI/CD pipelines and deployment workflows.
• Maintain and improve automation frameworks to accommodate evolving system architecture, including blockchain based components.
• Collaborate with developers, product managers, and DevOps engineers to ensure testability and quality across distributed systems.
• Create and manage defect workflows integrated with tools such as JIRA, ensuring clear communication and traceability.
• Define, track, and publish meaningful QA metrics to support automated test development, execution, and overall quality reporting.
• Build and maintain performance and load testing suites for critical backend services and smart contract interactions where applicable.
• Perform all types of testing as needed like Functional, API, Automation, Performance, and Integration testing.
• Contribute to continuous improvement of QA processes, tooling, and best practices within an Agile environment.

Qualifications

• 10+ years of experience with QA engineering delivery experience with strong focus on backend or API testing
• 6+ years of delivery within Agile SDLC teams ideally with CI/CD Experience testing AI/ML or LLM?powered systems, including model validation, prompt testing, regression testing of AI behavior, and evaluation of non?deterministic outputs
• Familiar with AI testing techniques and metrics such as precision and recall, drift detection, bias assessment, and offline vs online evaluation approaches
• Working knowledge of AI?enabled development tools or platforms (i.e. AI coding assistants, test?generation tools, or internal AI platforms) and their impact on SDLC and quality governance
• Strong experience testing REST APIs, backend microservices, and distributed systems
• Experience working with blockchain technologies (i.e. smart contracts, distributed ledgers, blockchain node interaction) is preferred
• Solid understanding of CI/CD tooling and pipelines, especially GitLab CI/CD
• Experience with Python, Behave, PyTest, Postman/REST Tools, JMeter or similar performance tools
• Experience with Gitlab, Git, Jenkins, and Docker
• AI Integration experience is preferred
• Exposure to banking or finance environments is preferred

Kavaliro provides Equal Employment Opportunities to all employees and applicants. All qualified applicants will receive consideration for employment without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.

Kavaliro is committed to the full inclusion of all qualified individuals.

In keeping with our commitment, Kavaliro will take the steps to assure that people with disabilities are provided reasonable accommodations. Accordingly, if reasonable accommodation is required to fully participate in the job application or interview process, to perform the essential functions of the position, and/or to receive all other benefits and privileges of employment, please respond to this posting to connect with a company representative.