Senior Data Quality Engineer, SDET with Databricks and Microsoft Fabric

DataArt Poland sp. z o.o.; Warszawa; On-site; PLN 120,000 - 190,000; Full time

Job summary

DataArt Poland sp. z o.o. is seeking a seasoned Quality Engineer focused on data testing and validation.

You will design and implement QA frameworks for data pipelines and products, ensuring source-to-target reconciliation and robust regression packs. The role emphasizes automated tests within CI/CD and handling schema drift to protect release quality. The project migrates data from legacy systems to a modern platform built on Databricks and Microsoft Fabric.

Qualifications

• 5+ years of experience in quality engineering.
• Strong Python and SQL skills, used for building test automation.
• Experience with pipeline validation, including row counts, control totals, referential integrity, and business rule assertions.
• Experience with source to target reconciliation using defined tolerances.
• Experience with test automation within CI/CD, including gates that block a release.
• Experience with Azure DevOps.
• Experience with SQL Server.
• English proficiency at a level suitable for direct client conversations.

Responsibilities

• Design and implement data testing and quality assurance frameworks.
• Build automated validation for data pipelines and data products.
• Own source to target reconciliation between legacy systems and the new platform.
• Build repeatable regression packs that run unattended.
• Write unit tests that execute automatically within CI/CD, with quality gates that can block a release.
• Detect and handle schema drift and unexpected structural changes.
• Validate reports, dashboards, and semantic models, including measure definitions, filter behaviour, and totals.
• Apply AI assisted techniques to test generation and coverage analysis, with human review.

Skills

Quality engineering

Python

SQL

CI/CD testing

Azure DevOps

SQL Server

English communication

Tools

Azure Databricks

dbt

Great Expectations

Databricks

Soda

Deequ

Job description

Nasze wymagania:

• 5+ years of experience in quality engineering, with meaningful recent experience testing data or ETL processes rather than applications.
• Strong Python and SQL skills, used for building test automation rather than only running it.
• Experience with pipeline validation, including row counts, control totals, referential integrity, and business rule assertions.
• Experience with source to target reconciliation using defined tolerances.
• Experience with test automation within CI/CD, including gates that block a release.
• Experience with Azure DevOps.
• Experience with SQL Server.
• English proficiency at a level suitable for direct client conversations.

Mile widziane:

• Experience with Azure Databricks or Microsoft Fabric.
• Experience with data quality frameworks, including Great Expectations, dbt tests, Databricks Expectations, DLT quality rules, Soda, or Deequ.
• Experience designing a test framework from scratch rather than extending an existing one.
• Experience testing semantic models and BI layers.
• Experience working in a regulated financial services environment.
• Experience with application test automation alongside data testing, including Selenium, C#, SpecFlow, or Playwright.

O projekcie:

The project migrates data and reporting from a legacy estate onto a modern platform built on Databricks and Microsoft Fabric. Quality engineering is central to the project. The business needs confidence that migrated data reconciles with the source, that pipelines behave as specified, and that reports produce consistent results.

Zakres obowiązków:

• Design and implement data testing and quality assurance frameworks.
• Build automated validation for data pipelines and data products.
• Own source to target reconciliation between legacy systems and the new platform.
• Build repeatable regression packs that run unattended.
• Write unit tests that execute automatically within CI/CD, with quality gates that can block a release.
• Detect and handle schema drift and unexpected structural changes.
• Validate reports, dashboards, and semantic models, including measure definitions, filter behaviour, and totals.
• Apply AI assisted techniques to test generation and coverage analysis, with human review.