Senior QA & ML Tester – AI Platform

EPAM Systems; Poland; Hybrid; PLN 180,000 - 260,000; Full time

Benefits offered by this job

Hybrid by design

Remote within Poland

Work abroad up to 60 days annually

Employee stock purchase plan 15%

Referral bonuses up to $2,000

Job summary

EPAM Systems in Poland is seeking a Senior QA / ML Tester to own quality assurance for the Agent Evaluation Framework built on AWS AgentCore. You’ll design, implement, and maintain a functional test suite for AI agent evaluation pipelines, focusing on on-demand integration testing, online sampling accuracy, and multi-evaluator execution.

This hands-on, production-focused role sits at the intersection of software quality engineering and AI/ML system testing within an Agile team, contributing to

Qualifications

• 5+ years of production QA automation or ML/AI testing experience.
• Experience testing AI/LLM systems and evaluation pipelines.
• Strong Python test automation with pytest and mocking.
• Familiarity with AWS AgentCore Evaluation API and OpenTelemetry.
• CI/CD integration and Jira defect lifecycle experience.

Responsibilities

• Design and implement a functional test suite for the AWS AgentCore Evaluation API using pytest, covering known-good / known-bad session pair validation, multi-evaluator execution correctness, and edge case handling
• Develop and maintain integration tests for on-demand evaluation mode, integrated into the CI/CD pipeline with automated execution on each build
• Validate online mode sampling accuracy, design test scenarios, define acceptance criteria, and report deviations with reproducible evidence
• Conduct and document a feasibility assessment for non-AgentCore runtime evaluation: analyze alternative runtimes, define evaluation methodology, and deliver a structured findings report
• Test OpenTelemetry trace‑based evaluation inputs and validate ADOT trace ingestion, trace structure correctness, and evaluator input integrity
• Collaborate with platform engineers to clarify evaluation contracts, reproduce defects, and align on quality gates
• Maintain test documentation, including test plans, test reports, defect logs, and evaluation feasibility artifacts in Confluence/Jira
• Participate in Agile ceremonies, including sprint planning, daily standups, demos, and retrospectives
• Contribute to EngX practices such as code review of test scripts, CI/CD pipeline integration, and test coverage reporting

Skills

Python

pytest

API testing

CI/CD

AWS AgentCore

OpenTelemetry

Jira

AI/ML testing

English B2+

REST API security

Tools

GitHub Actions

Jenkins

Unittest.mock

moto

SigV4

Job description

EPAM Systems in Poland is seeking a Senior QA / ML Tester to own quality assurance for the Agent Evaluation Framework built on AWS AgentCore. You’ll design, implement, and maintain a functional test suite for AI agent evaluation pipelines, focusing on on-demand integration testing, online sampling accuracy, and multi-evaluator execution.

This hands-on, production-focused role sits at the intersection of software quality engineering and AI/ML system testing within an Agile team, contributing to