Senior Data QA - Databricks Migration & Validation

BuzzClan; Dallas (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

BuzzClan seeks an experienced Data/ETL Test Engineer to lead testing for migrating a SQL Server data platform to Databricks on Azure. You will design and implement data parity tests, validate transformations, and ensure pipelines run end-to-end with CI/CD integration.

Responsibilities include building automated reconciliations, validating Delta Lake features, and enforcing quality gates across releases. Strong SQL, PySpark, and Databricks experience are required, with a focus on production-ready

Qualifications

• 5+ years in data/ETL testing, including at least one data platform migration.
• Strong SQL (T-SQL and Spark SQL) with joins, window functions, diffs, and aggregates.
• Hands-on Databricks experience: notebooks, Jobs/Workflows, Delta Lake, Unity Catalog basics.
• Hands-on Azure Data Factory experience: pipelines, linked services, datasets, triggers, monitoring.

Responsibilities

• Define migration test strategy and plan across schema, data, transformations, and performance.
• Build automated reconciliation: row counts, checksums, diffs, and full row-level diffs for critical tables.
• Convert legacy SQL logic into test cases and validate PySpark/Spark SQL output.
• Define quality gates and release criteria; validate ADF pipelines end-to-end.

Skills

T-SQL

Spark SQL

Databricks

Azure Data Factory

PySpark

CI/CD Testing

Tools

Databricks

Azure DevOps

Job description

BuzzClan seeks an experienced Data/ETL Test Engineer to lead testing for migrating a SQL Server data platform to Databricks on Azure. You will design and implement data parity tests, validate transformations, and ensure pipelines run end-to-end with CI/CD integration.

Responsibilities include building automated reconciliations, validating Delta Lake features, and enforcing quality gates across releases. Strong SQL, PySpark, and Databricks experience are required, with a focus on production-ready