Senior AI Test Automation Architect (ML & MLOps)

UST; Madrid; Hybrid; EUR 60,000 - 90,000; Full time

Benefits offered by this job

Health Care Plan

Teleworking compensation

Life and Accident Insurances

Retribución Flexible Program

Training platforms access

Career stability and career plans

Referral bonuses

Flexible payments (12 or 14) per year

Work-life balance policies

UST Club platform discounts

Job summary

UST is seeking a Senior AI Test Automation Engineer to own end-to-end testing for an advanced AI/ML platform. The role focuses on designing scalable automation frameworks, integrating tests into CI/CD, and ensuring robust AI/ML test coverage across APIs and microservices.

The position is hybrid in Madrid, with 40 hours weekly and a comprehensive benefits program. Join a global leader in digital technology services and contribute to high-impact AI initiatives.

Qualifications

• Strong experience in Python-based test automation with Playwright/pytest/Behave (BDD).
• Ability to design and own test frameworks, not just scripts.
• Experience integrating automation into CI/CD pipelines (GitLab preferred).
• Knowledge of parallel execution, reporting tools, and flaky test management.
• Understanding of APIs, microservices, event-driven architectures.
• Familiarity with SAFe, Agile, and DevOps principles.
• Experience with Jira and test management tools (Xray, Agile Hive, etc.).

Responsibilities

• Define and evolve the test automation architecture aligned with the Bank’s Test Strategy.
• Design scalable, reusable automation frameworks and patterns across teams.
• Analyse requirements and ensure testability during Agile ceremonies.
• Design and automate test cases (functional & non-functional) based on risk and coverage.
• Implement automation using Python frameworks (Playwright, pytest, Behave/BDD).
• Develop AI/ML-specific testing approaches (LLM evaluation, non-deterministic behaviour, RAG workflows).
• Build and maintain API contract tests and microservices validation.
• Integrate and run automated tests in CI/CD pipelines (GitLab).
• Use Docker/Kubernetes for reproducible test environments.
• Manage test execution across environments (DEV, INT, UAT, pre-PROD).
• Monitor test health, handle flaky tests, and ensure reliable reporting.
• Report defects in Jira with proper traceability and evidence.
• Produce audit-ready test artefacts and ensure governance compliance.
• Collaborate with cross-functional teams (POs, Engineers, MLOps, Data Engineers).
• Act as Test Automation Ambassador within the ART.
• Ensure end-to-end traceability across requirements, tests, and results.
• Maintain and evolve test suites as systems change.

Skills

Python

Playwright

pytest

BDD frameworks

GitLab CI

Docker

Kubernetes

APIs testing

Jira

QA in AI/ML

Tools

GitLab CI

Job description

UST is seeking a Senior AI Test Automation Engineer to own end-to-end testing for an advanced AI/ML platform. The role focuses on designing scalable automation frameworks, integrating tests into CI/CD, and ensuring robust AI/ML test coverage across APIs and microservices.

The position is hybrid in Madrid, with 40 hours weekly and a comprehensive benefits program. Join a global leader in digital technology services and contribute to high-impact AI initiatives.