Senior AI Test Automation Engineer (Hybrid, Madrid)

UST; Spain; Hybrid; EUR 70,000 - 110,000; Full time

Benefits offered by this job

Annual Leave 23d

Discretionary days

Health Care Plan

Teleworking allowance

Life Insurance

Training Access

Career stability

Referral bonus

12/14 payments

Work-life balance

Club discounts

Gym discounts

Job summary

UST is seeking a Senior AI Test Automation Engineer to lead the test automation strategy for an advanced AI & MLOps platform. You will design scalable frameworks and automate tests across APIs, UIs, data pipelines, and AI/ML components such as LLMs and RAG systems.

You will own end-to-end automation ownership, work in Agile/SAFe environments, and focus on quality engineering, reliability, and automation at scale. Hybrid Madrid with client-site days.

Qualifications

• 5–7+ years of experience in test automation using Python.
• Experience owning automation frameworks, not just writing tests.
• Solid experience with Playwright, pytest, API testing, and BDD frameworks.
• Experience integrating automation into CI/CD pipelines (GitLab or similar).
• Knowledge of Docker and basic Kubernetes concepts.
• Exposure to AI/ML systems or strong interest in testing AI-driven applications.
• Experience with Jira and test management tools (e.g., Xray).
• Strong communication, ownership, and collaboration skills.
• Awareness of security, data privacy in regulated environments.

Responsibilities

• Own the design, architecture, and evolution of the test automation framework for the AI & MLOps platform.
• Define and implement scalable, reusable automation patterns (UI, API, data, BDD).
• Automate test cases using Python stack (Playwright, pytest, requests, pytest-bdd/Behave) applying clean code practices.
• Design and execute AI/ML-specific tests (e.g., LLM evaluation, RAG validation, non-deterministic outputs).
• Integrate and orchestrate automated tests within CI/CD pipelines (GitLab).
• Provision and manage test environments using Docker and Kubernetes.
• Plan and execute regression, smoke, and release test suites across DEV, INT, UAT, and pre-PROD.
• Monitor execution quality, manage flaky tests, and ensure trustworthy test results.
• Analyze defects, perform root cause analysis, and report issues with clear evidence in Jira.
• Ensure full traceability from requirements to test execution and reporting.
• Collaborate with cross-functional teams (ML Engineers, MLOps, Data Engineers, Product Owners).
• Act as Test Automation Ambassador, promoting best practices and guiding teams on tooling and strategy.
• Ensure compliance with test policies, audit requirements, and data privacy standards.

Skills

Python automation

Playwright

pytest

API testing

BDD frameworks

CI/CD pipelines

GitLab

Docker

Kubernetes basics

Agile/SAFe

Jira

Xray

Communication

Tools

GitLab CI

Jira

Xray

Job description

UST is seeking a Senior AI Test Automation Engineer to lead the test automation strategy for an advanced AI & MLOps platform. You will design scalable frameworks and automate tests across APIs, UIs, data pipelines, and AI/ML components such as LLMs and RAG systems.

You will own end-to-end automation ownership, work in Agile/SAFe environments, and focus on quality engineering, reliability, and automation at scale. Hybrid Madrid with client-site days.