Senior Automation Tester in Python (for Gen AI powered applications)

EPAM Systems Inc; United States; On-site; USD 120,000 - 160,000; Full time

Job summary

EPAM Systems is seeking a Senior Automation Tester in Python to join Production Pods. This role focuses on automated test frameworks for AI/GenAI-powered clinical applications, validating non-deterministic AI outputs, and ensuring GxP compliance through IQ/OQ/PQ qualification.

You will design tests across Safety, Regulatory, Clinical Operations, and SAGE platform tooling, and collaborate with data scientists and backend developers to define acceptance criteria.

Qualifications

• 5+ years hands-on QA/test automation experience.
• Experience building automated test suites from scratch for AI-powered apps.
• Experience with AI/GenAI validation, GxP awareness, IQ/OQ/PQ.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical apps using Playwright, Selenium, and pytest.
• Design and execute test strategies for non-deterministic AI outputs, including accuracy and prompt regression.
• Integrate automated tests into CI/CD pipelines with gatekeeping and coverage thresholds.
• Execute GxP-compliant IQ/OQ/PQ scripts and compile qualification evidence packages.
• Manage AI-specific test data, including edge cases and golden datasets.
• Monitor model drift and performance over time; document test plans and evidence.
• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria.

Skills

Playwright

Selenium

Cypress

pytest

Python

CI/CD

API testing

REST APIs

Tools

Jira/Xray

TestRail

Zephyr

Postman/Newman

Jenkins

GitHub Actions

Job description

We are looking for a Senior Automation Tester in Python to join Production Pods within the program. This is not a manual/functional QA role. You will build and maintain automated test frameworks for AI/GenAI-powered clinical applications, validate non-deterministic AI outputs, and ensure GxP compliance through automated IQ/OQ/PQ qualification. You will work across multiple use cases in Safety, Regulatory, Clinical Operations, and SAGE platform tooling.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest
• Design and execute test strategies for non-deterministic AI/GenAI outputs, including accuracy, RAG relevance, hallucination checks, and prompt regression
• Integrate automated tests into CI/CD pipelines with quality gates such as block-on-failure and coverage thresholds
• Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages
• Manage AI-specific test data, including edge cases, adversarial inputs, and golden datasets
• Monitor model drift, regression, and performance degradation over time
• Produce test strategy documents, test plans, and compliance-ready test evidence
• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features

Requirements

• 5+ years of hands-on QA/test automation experience
• Expertise in building and maintaining automated test suites from scratch using Playwright, Selenium, Cypress, and pytest
• Skills in API and backend test automation, including REST APIs, microservices, and serverless architectures with Postman/Newman and pytest + requests
• Proficiency in Python as the primary automation language, with the ability to write harnesses, fixtures, and utilities independently
• Knowledge of CI/CD test integration, embedding automated tests into pipelines such as Jenkins and GitHub Actions with quality gates
• Background in testing AI/GenAI applications, validating non-deterministic AI/LLM outputs for accuracy, RAG relevance, hallucination, and prompt regression
• Familiarity with AI/ML model testing, including accuracy, bias, and performance validation using RAGAS, LLM-as-judge, and golden datasets
• English proficiency at B2 level or higher
• Nice to have Background in pharmaceutical / life sciences domain, with GxP awareness and IQ/OQ/PQ qualification
• Experience with Jira/Xray, TestRail, or Zephyr test management
• Skills in performance/load testing using JMeter and Locust
• Understanding of security testing basics such as OWASP
• Familiarity with AWS services including Lambda, API Gateway, S3, and CloudWatch