Senior Data QA Engineer: Cloud ETL & Databricks Migration

BuzzClan; Plano (TX); On-site; USD 110,000 - 140,000; Full time

Job summary

BuzzClan is seeking a Senior Data QA / Test Engineer to lead testing for migrating a SQL Server data platform to Databricks on Azure. You will validate data parity, establish quality gates, and oversee end-to-end validation of ADF pipelines and Delta Lake workloads.

The role requires 5+ years in Data/ETL Testing, strong SQL, PySpark, and CI/CD experience, plus expertise in data migrations and test automation. Contract-to-fulltime W2 options available in Plano, TX.

Qualifications

• Must have strong experience in Data/ETL testing and data validation.
• Ability to design test cases from data migration requirements.
• Experience with cloud data platforms and SQL proficiency.

Responsibilities

• Define and execute the migration test strategy and test plan covering schema, data, transformation logic, performance, and regression testing
• Build automated source-to-target reconciliation for migrated tables
• Validate row counts, column-level checksums/hashes, aggregates, and row-level differences
• Convert legacy SQL business logic, stored procedures, and views into test cases
• Validate equivalent PySpark/Spark SQL outputs against legacy SQL results
• Define and enforce quality gates across code merge, Bronze/Silver/Gold promotion, UAT sign-off, and production cutover
• Validate Azure Data Factory pipelines including parameters, triggers, dependencies, mappings, failure/retry paths, incremental loads, and watermark logic
• Test Delta Lake behavior including MERGE/upsert, SCD Type 1/2, schema evolution, deduplication, and late-arriving data
• Integrate automated tests into CI/CD pipelines using Azure DevOps, GitHub Actions, or equivalent
• Partner with the Data Quality Engineer to embed DQX/data-quality checks into pipelines
• Report defects, reconciliation results, test metrics, and go/no-go readiness to project leadership
• Support UAT, regression testing, production validation, defect resolution, and migration cutover activities
• Ensure zero critical data defects are introduced into production following each migration wave

Skills

Data QA / ETL Testing

SQL / T-SQL / Spark SQL

PySpark

Data Migration Testing

Source-to-Target Reconciliation

Data Validation / Quality Testing

Test Automation / pytest

CI/CD / Azure DevOps or GitHub Actions

Quality Gates / Test Strategy

Tools

Databricks

Delta Lake

Azure Data Factory (ADF)

Databricks Notebooks / Jobs

Unity Catalog

Azure DevOps

GitHub Actions

Job description

BuzzClan is seeking a Senior Data QA / Test Engineer to lead testing for migrating a SQL Server data platform to Databricks on Azure. You will validate data parity, establish quality gates, and oversee end-to-end validation of ADF pipelines and Delta Lake workloads.

The role requires 5+ years in Data/ETL Testing, strong SQL, PySpark, and CI/CD experience, plus expertise in data migrations and test automation. Contract-to-fulltime W2 options available in Plano, TX.