Senior Data QA

BuzzClan; Dallas (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

BuzzClan seeks an experienced Data/ETL Test Engineer to lead testing for migrating a SQL Server data platform to Databricks on Azure. You will design and implement data parity tests, validate transformations, and ensure pipelines run end-to-end with CI/CD integration.

Responsibilities include building automated reconciliations, validating Delta Lake features, and enforcing quality gates across releases. Strong SQL, PySpark, and Databricks experience are required, with a focus on production-ready

Qualifications

• 5+ years in data/ETL testing, including at least one data platform migration.
• Strong SQL (T-SQL and Spark SQL) with joins, window functions, diffs, and aggregates.
• Hands-on Databricks experience: notebooks, Jobs/Workflows, Delta Lake, Unity Catalog basics.
• Hands-on Azure Data Factory experience: pipelines, linked services, datasets, triggers, monitoring.

Responsibilities

• Define migration test strategy and plan across schema, data, transformations, and performance.
• Build automated reconciliation: row counts, checksums, diffs, and full row-level diffs for critical tables.
• Convert legacy SQL logic into test cases and validate PySpark/Spark SQL output.
• Define quality gates and release criteria; validate ADF pipelines end-to-end.

Skills

T-SQL

Spark SQL

Databricks

Azure Data Factory

PySpark

CI/CD Testing

Tools

Databricks

Azure DevOps

Job description

We are migrating our SQL Server data platform (stored procedures, SSIS/SQL jobs, reporting tables) to Databricks on Azure, orchestrated by Azure Data Factory. In this role, you will own the testing strategy for this migration: proving data parity between legacy SQL and Databricks, defining the quality gates each release must pass, and validating ADF pipelines end-to-end.

Key Responsibilities:

• Define the migration test strategy and plan covering schema, data, transformation logic, performance, and regression testing.
• Build automated source-to-target reconciliation: row counts, column-level checksums/hashes, aggregates, and full row-level diffs for critical tables.
• Convert legacy SQL business logic (stored procedures, views) into test cases and validate equivalent PySpark/Spark SQL output.
• Define and enforce quality gates across the lifecycle: code merge, Bronze/Silver/Gold layer promotion, UAT sign-off, and production cutover.
• Validate ADF pipelines: parameters, triggers, dependencies, copy activity mappings, failure/retry paths, incremental loads, and watermark logic.
• Test Delta Lake behavior: MERGE/upsert, SCD Type 1/2, schema evolution, deduplication, and late-arriving data.
• Integrate tests into CI/CD (Azure DevOps or GitHub Actions) with Databricks Asset Bundles or equivalent.
• Report defects, reconciliation results, and go/no-go readiness to project leadership.
• Partner with the Data Quality Engineer to embed Databricks Labs DQX checks into pipelines.
• Reconciliation framework running automatically for all migrated tables.
• Quality gates documented, agreed, and enforced in the release pipeline.
• Zero critical data defects found in production after each migration wave.

Must-Have Skills:

• 5+ years in data/ETL testing, including at least one data platform migration.
• Strong SQL (T-SQL and Spark SQL): joins, window functions, EXCEPT/MINUS-based diffs, and aggregates.
• Hands-on Databricks experience: notebooks, Jobs/Workflows, Delta Lake, and Unity Catalog basics.
• Hands-on Azure Data Factory experience: pipelines, linked services, datasets, triggers, monitoring, and debugging runs.
• PySpark for building reusable test and reconciliation frameworks.
• Experience defining quality gates, entry/exit criteria, and test metrics.
• Test automation with pytest or similar, integrated into CI/CD.

Nice-to-Have Skills:

• Databricks Labs DQX, Great Expectations, Soda, or Lakeflow (DLT) expectations.
• Databricks Asset Bundles, Terraform, and Azure DevOps.
• Performance testing of Spark jobs (partitioning, file sizes, cluster sizing).
• Knowledge of SSIS, SSRS, or Power BI report validation.