AI Test Automation Engineer 100% (f/m/d)

Juliusbaer

Madrid

On-site

EUR 60,000 - 100,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Julius Baer is seeking a Senior Test Automation Engineer to lead the automation strategy for the ML & AI ART. You will own architecture and hands-on delivery of tests for APIs, UIs, data pipelines, and AI/ML components, collaborating with ML engineers, data engineers, and product teams.

You will design scalable patterns, implement tests with Python, and drive QA in GitLab CI/CD environments, ensuring traceability, reproducibility, and audit-ready artefacts across dev-to-prod environments.

Qualifications

  • Proven expertise in Python-based test automation and framework ownership.
  • Hands-on experience in CI/CD pipelines and test data management.
  • Experience with UI and API automation using Playwright-Python and pytest.
  • Strong knowledge of BDD/Gherkin and contract testing concepts.

Responsibilities

  • Define and evolve the technical test automation approach and framework architecture.
  • Automate identified test cases using Python-based frameworks and maintain test data builders.
  • Integrate automated tests in GitLab CI/CD pipelines and ensure reliable test execution.
  • Design and implement AI/ML-specific test cases for LLM outputs and RAG workflows.
  • Maintain contract tests (e.g., Pact) for API boundaries and microservices.
  • Collaborate with Product Owners, ML/AI engineers, and QA teams for test strategy alignment.

Skills

Python-based testing
Playwright-Python
Gherkin/BDD
CI/CD
API testing
Docker
Kubernetes basics
Jira QA integration
Test framework design
English fluency

Education

BSc or MSc in CS/Engineering

Tools

Git
Allure
pytest
Xray

Job description

At Julius Baer, we celebrate and value the individual qualities you bring, enabling you to be impactful, to be entrepreneurial, to be empowered, and to create value beyond wealth. Let`s shape the future of wealth management together. You will join our ML & AI ART as a Senior Test Automation Engineer, responsible for the technical test automation solution supporting our AI and MLOps platform. The platform powers AI-driven capabilities across the Bank including RAG systems, LLM-based assistants, agentic workflows, and ML pipelines serving business-critical use cases. You will own the architecture, implementation, and execution of our test automation frameworks, covering both traditional software components (APIs, UIs, data pipelines) and emerging AI/ML-specific behaviour, and act as the Test Automation Ambassador within the ART. You will work within the Bank's Test Strategy, while owning the technical depth and hands-on delivery of automation for your platform.

YOUR CHALLENGE

Define and evolve the technical test automation approach and framework architecture for the ML & AI ART, aligned with the Bank's Test Strategy and Test Policy

Design reusable, scalable test automation patterns (page objects, API clients, test data builders) that other engineers across squads can adopt, ensuring technical consistency of test automation across the ART

Analyse and evaluate requirements, Features, and Stories for testability during PI Planning, Backlog Refinement, and Iteration Planning

Derive test cases from technical and risk analysis of both functional and non-functional requirements (reliability, performance, security, usability, robustness), selecting appropriate test techniques and automation scope based on risk, coverage goals, and ROI

Automate identified test cases using Python-based frameworks – Playwright-Python for UI, requests + pytest for APIs, Behave or pytest-bdd for BDD/Gherkin – applying clean code principles, reusability, readability, and stability

Design and implement AI/ML-specific test cases: evaluation pipelines for LLM outputs

Build and maintain contract tests (e.g. Pact) for platform APIs and microservice boundaries

Integrate and orchestrate automated tests in GitLab CI/CD pipelines, including merge request pipelines, GitLab runners

Leverage Docker and Kubernetes to provision isolated, reproducible test environments

Plan, schedule, and trigger automated test executions across environments (DEV, INT, UAT, pre-PROD) – including regression suites, smoke tests, release executions, and on-demand runs tied to merge requests and PI milestones

Monitor execution health, investigate and quarantine flaky tests, and maintain a low false-positive rate to keep quality signals trustworthy

Triage execution results, raise defects in Jira with evidence (logs, traces, screenshots, videos), and communicate quality signals to the squad, Product Owner, and Test Manager

Produce execution evidence (run metadata, artefacts, reports) suitable for audit and release governance, in line with the Bank's Test Policy and retention requirements

Contribute actively to PI Planning, System Demos, Inspect & Adapt, and other SAFe ceremonies as part of the ML & AI ART

Ensure end-to-end traceability from Jira Features and Stories › automated tests › defects › test results, leveraging the Bank's test management integration

Author and maintain BDD scenarios in Gherkin, linked to acceptance criteria on Stories

Collaborate closely with Product Owners, Scrum Masters, ML Engineers, MLOps Engineers, and Data Engineers across squads

Align work with the Bank's Test Strategy, and reporting requirements, supporting audit-ready artefacts, traceability matrices, and test progress reporting

Support root cause analysis of defects using logs and traces

Act as Test Automation Ambassador within the ML & AI ART consulting squads on automation design, framework usage, tooling choices, and test data strategy

Prepare test data, ensuring synthetic or anonymised data is used wherever possible to meet confidentiality expectations

Design AI/ML-specific test datasets where needed

Own test maintenance and refactoring in response to UI, API, model, or prompt changes, ensuring continued compliance with access controls and test environment policies

YOUR PROFILE

Proven expertise in Python-based test automation: Playwright-Python (UI), Behave or pytest-bdd (Gherkin/BDD), requests + pytest (API/service validation practical equivalence to REST-Assured)

Demonstrated ability to design and own test automation frameworks, not just write test scripts including reusable utilities and maintainability patterns

Hands-on experience integrating and executing automated tests within CI/CD pipelines, ideally GitLab or equivalent enterprise platforms

Experience with distributed test execution, parallelisation, flaky-test management, and modern reporting tools (Allure, pytest-html, or equivalent)

Solid grasp of Git and version control workflows, clean code principles, and code review culture

Working knowledge of Docker; familiarity with Kubernetes basics (jobs, namespaces)

Exposure to testing AI/ML systems, or strong motivation to develop this expertise: evaluation of LLM outputs, handling non-deterministic responses, evals for RAG and agentic workflows

Understanding of API design, microservices, event-driven architectures, and authentication layers

Sound understanding of SAFe and DevOps principles; experience operating in an Agile Release Train is a plus

Experience with Jira for Story/Feature tracking and test management integration (Xray, Agile Hive)

Demonstrated end-to-end thinking – connecting user journeys, data flows, authentication layers, and system boundaries

Comfortable working within an established Test Strategy, collaborating with Test Managers on execution planning, reporting, and compliance

Demonstrated awareness of information security, data privacy, and compliant test data handling in regulated environments

Collaborative team player with strong ownership takes automation problems from analysis to execution to resolution with minimal supervision

Strong communication skills, able to work effectively with engineers, Product Owners, Scrum Masters, Architects, and Test Managers

Strong organisational skills, structured, reliable

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Minimum 5-7 years in Test Automation with substantial hands-on Python experience, including demonstrated framework design, ownership, and test execution at scale (not only script-level contributions)

Experience in a regulated environment (financial services, healthcare, pharma) strongly preferred

Certifications in ISTQB (Foundation as baseline) SAFe (SP, SSM, or equivalent), or DevOps disciplines are a plus

Exposure to AI/ML systems through testing, development, or applied projects is a strong plus; appetite to develop deep AI/ML testing expertise is essential

Fluency in English; German is a plus

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Test Automation Engineer 100% (f/m/d)
AI Test Automation Engineer 100% (f/m/d)

Julius Baer Group • Madrid

On-site
EUR 55,000 - 75,000
Senior Test Automation Engineer 100% (f/m/d)
Senior Test Automation Engineer 100% (f/m/d)

Bank Julius Bär & Co. Ltd. • Madrid

On-site
EUR 85,000 - 110,000
Lead Test Framework Architect (M/F/D)
Lead Test Framework Architect (M/F/D)

Agilent • Spain

On-site
EUR 110,000 - 140,000
Senior QA Automation Engineer 100% (f/m/d)
Senior QA Automation Engineer 100% (f/m/d)

Julius Baer Group • Madrid

On-site
EUR 45,000 - 65,000
Senior Test Automation AI Engineer
Senior Test Automation AI Engineer

EPAM Systems • Spain

Hybrid
PHP 5,084,000 - 7,988,000
Test Automation Platform Engineer (80-100%) (F/M/D)
Test Automation Platform Engineer (80-100%) (F/M/D)

Julius Baer Group • Madrid

On-site
EUR 55,000 - 85,000
Senior QA Automation Engineer 100% (f/m/d)
Senior QA Automation Engineer 100% (f/m/d)

Julius Baer Group • Madrid

On-site
EUR 60,000 - 90,000
Senior AI/ML Test Automation Engineer
Senior AI/ML Test Automation Engineer

Julius Baer Group • Madrid

On-site
EUR 55,000 - 75,000
Senior QA Test Automation Engineer
Senior QA Test Automation Engineer

Dematic Corp. (ILD-US) • Barcelona

Hybrid
EUR 70,000 - 95,000
Private health insurance
Food Allowance
Hybrid Work Model
+2
Senior AI/ML Test Automation Engineer
Senior AI/ML Test Automation Engineer

Juliusbaer • Madrid

On-site
EUR 60,000 - 100,000