Senior QA Engineer – ML/AI Evaluation Pipelines

EPAM Systems; Poland; Hybrid; PLN 180,000 - 260,000; Full time

Benefits offered by this job

Hybrid work design

Work remotely within Poland

Opportunity to work abroad up to 60/90

Job summary

EPAM Systems is seeking a Senior QA / ML Tester to own quality assurance for the Agent Evaluation Framework built on AWS AgentCore. You will design, implement, and maintain a functional test suite for AI/ML evaluation pipelines, ensuring reliability in an Agile team.

The role requires hands-on testing across on-demand and online modes, OpenTelemetry integration validation, and defect reporting with Jira. Hybrid work in Poland is available, with potential for travel and collaboration with

Qualifications

• 5+ years of production QA automation or ML/AI system testing.
• Experience testing AI/LLM evaluation pipelines and agent behavior.
• Proficiency in Python test automation with pytest, fixtures, parametrize, and mocking.
• Knowledge of AWS AgentCore Evaluation API for on-demand and online modes.
• Familiarity with OpenTelemetry/ADOT for trace-based testing.
• REST API testing, including request/response validation and authentication.
• CI/CD integration using GitHub Actions/Jenkins.
• Test data management with known-good/known-bad session pairs.
• Defect lifecycle management with Jira.
• Understanding of LLM/agent evaluation concepts and evaluation contracts.
• English at least B2+ for collaboration.

Responsibilities

• Design and implement a functional test suite for AWS AgentCore Evaluation API using pytest.
• Develop and maintain integration tests for on-demand evaluation, CI/CD integrated.
• Validate online mode sampling accuracy; design scenarios and acceptance criteria.
• Assess non-AgentCore runtime evaluation feasibility and deliver findings.
• Test OpenTelemetry trace-based evaluation inputs and ADOT trace ingestion.
• Collaborate with platform engineers to clarify evaluation contracts and reproduce defects.
• Maintain test documentation in Confluence/Jira.
• Participate in Agile ceremonies and EngX practices such as code reviews.
• Contribute to test coverage reporting and CI/CD improvements.

Skills

Python testing

pytest

REST API testing

CI/CD integration

OpenTelemetry testing

AWS AgentCore

ML/AI system testing

Jira bug reports

Tools

GitHub Actions

Jenkins

unittest.mock

moto

OpenTelemetry

Job description

EPAM Systems is seeking a Senior QA / ML Tester to own quality assurance for the Agent Evaluation Framework built on AWS AgentCore. You will design, implement, and maintain a functional test suite for AI/ML evaluation pipelines, ensuring reliability in an Agile team.

The role requires hands-on testing across on-demand and online modes, OpenTelemetry integration validation, and defect reporting with Jira. Hybrid work in Poland is available, with potential for travel and collaboration with