BigData Test Automation Specialist

EPAM Systems; Schweiz; On-site; EUR 117,000 - 159,000; Full time

Job summary

EPAM Systems is seeking a BigData Test Automation Specialist in Zurich to design and implement robust testing strategies for a cutting-edge big data platform in the banking sector. You will work with Trino, Hive, Kafka, and Dataiku to ensure data quality, platform resilience, and system integrity.

Responsibilities include building scalable test frameworks, data quality checks, and CI/CD automation to support streaming pipelines and complex SQL transformations.

Qualifications

• Strong experience with Python and the Pytest framework for test automation.
• Hands-on experience with big data technologies like Trino, Hive, Kafka or S3.
• Proficiency with data quality and validation tools.
• Experience with containerization and mocking tools like Docker, Testcontainers.
• Proven ability to test streaming applications and event-driven architectures.
• Knowledge of scripting for cloud object storage interaction.
• Experience with MLOps platforms, preferably Dataiku.
• Proven experience building and managing CI/CD pipelines (e.g. GitHub Actions, GitLab CI).
• Familiarity with SQL transformation tools like dbt or SQLMesh.
• Understanding of data versioning concepts.
• Fluent in English.
• Nice to have Experience in the banking or financial services sector.

Responsibilities

• Design and implement comprehensive test automation strategies for a big data platform
• Develop and maintain scalable test frameworks
• Implement declarative data quality checks for schemas
• Create and manage automated tests for complex SQL transformations
• Test and validate Hive metastore schemas and data integrity
• Develop tests for streaming data pipelines using Kafka and Testcontainers
• Automate Dataiku DSS scenarios and integrate them into CI/CD pipelines
• Build and optimize CI/CD infrastructure for efficient big data testing
• Perform resilience and chaos testing to ensure platform stability
• Use data versioning patterns like Write-Audit-Publish for isolated and safe testing

Skills

Python

Pytest

Streaming tests

Data quality

SQL testing

CI/CD pipelines

Dataiku

GitHub Actions

GitLab CI

English

Banking domain

Tools

Docker

Testcontainers

dbt

SQLMesh

GitHub Actions

GitLab CI

Job description

We are looking for a BigData Test Automation Specialist to join our team in Zurich. This role focuses on designing and implementing a robust, multi-layered testing strategy for a modern big data platform within the banking sector. You will work with cutting-edge technologies including Trino, Kafka, Iceberg and Dataiku to ensure data quality, platform resilience and overall system integrity.

Responsibilities

• Design and implement comprehensive test automation strategies for a big data platform
• Develop and maintain scalable test frameworks
• Implement declarative data quality checks for schemas
• Create and manage automated tests for complex SQL transformations
• Test and validate Hive metastore schemas and data integrity
• Develop tests for streaming data pipelines using Kafka and Testcontainers
• Automate Dataiku DSS scenarios and integrate them into CI/CD pipelines
• Build and optimize CI/CD infrastructure for efficient big data testing
• Perform resilience and chaos testing to ensure platform stability
• Use data versioning patterns like Write-Audit-Publish for isolated and safe testing

Requirements

• Strong experience with Python and the Pytest framework for test automation
• Hands-on experience with big data technologies like Trino, Hive, Kafka or S3
• Proficiency with data quality and validation tools
• Experience with containerization and mocking tools like Docker, Testcontainers
• Proven ability to test streaming applications and event-driven architectures
• Knowledge of scripting for cloud object storage interaction
• Experience with MLOps platforms, preferably Dataiku
• Proven experience building and managing CI/CD pipelines (e.g. GitHub Actions, GitLab CI)
• Familiarity with SQL transformation tools like dbt or SQLMesh
• Understanding of data versioning concepts
• Fluent in English
• Nice to have Experience in the banking or financial services sector