Senior ETL Migration QA Engineer (Databricks/Azure)

BuzzClan; Plano (TX); On-site; USD 120,000 - 170,000; Full time

Job summary

BuzzClan is migrating its SQL Server data platform to Databricks on Azure. You will own the testing strategy, validate parity between legacy SQL and Databricks, and define quality gates for releases across the CI/CD pipeline.

You will build automated reconciliation, convert legacy logic to PySpark test cases, and validate ADF pipelines, Delta Lake behaviour, and data quality checks. This role requires strong SQL skills and Databricks experience in a hands-on, collaborative environment.

Qualifications

• 5+ years in data/ETL testing.
• Strong SQL (T-SQL and Spark SQL).
• Hands-on Databricks: notebooks, Jobs/Workflows, Delta Lake, Unity Catalog basics.
• Hands-on Azure Data Factory: pipelines, linked services, datasets, triggers, monitoring and debugging runs.
• PySpark for building reusable test and reconciliation frameworks.
• Experience defining quality gates, entry/exit criteria and test metrics.
• Test automation with pytest or similar, integrated into CI/CD.

Responsibilities

• Reconciliation framework running automatically for all migrated tables.
• Quality gates documented, agreed and enforced in the release pipeline.
• Zero critical data defects found in production after each migration wave.

Skills

ETL testing

SQL

Quality gates

CI/CD

Data platform migration

pytest

Test automation

Tools

Databricks

Azure Data Factory

PySpark

Delta Lake

Unity Catalog

CI/CD tooling

Job description

BuzzClan is migrating its SQL Server data platform to Databricks on Azure. You will own the testing strategy, validate parity between legacy SQL and Databricks, and define quality gates for releases across the CI/CD pipeline.

You will build automated reconciliation, convert legacy logic to PySpark test cases, and validate ADF pipelines, Delta Lake behaviour, and data quality checks. This role requires strong SQL skills and Databricks experience in a hands-on, collaborative environment.