DATA QUALITY ENGINEER

BuzzClan; Plano (TX); On-site; USD 110,000 - 140,000; Full time

Job summary

BuzzClan is seeking a Senior Data QA / Test Engineer to lead testing for migrating a SQL Server data platform to Databricks on Azure. You will validate data parity, establish quality gates, and oversee end-to-end validation of ADF pipelines and Delta Lake workloads.

The role requires 5+ years in Data/ETL Testing, strong SQL, PySpark, and CI/CD experience, plus expertise in data migrations and test automation. Contract-to-fulltime W2 options available in Plano, TX.

Qualifications

• Must have strong experience in Data/ETL testing and data validation.
• Ability to design test cases from data migration requirements.
• Experience with cloud data platforms and SQL proficiency.

Responsibilities

• Define and execute the migration test strategy and test plan covering schema, data, transformation logic, performance, and regression testing
• Build automated source-to-target reconciliation for migrated tables
• Validate row counts, column-level checksums/hashes, aggregates, and row-level differences
• Convert legacy SQL business logic, stored procedures, and views into test cases
• Validate equivalent PySpark/Spark SQL outputs against legacy SQL results
• Define and enforce quality gates across code merge, Bronze/Silver/Gold promotion, UAT sign-off, and production cutover
• Validate Azure Data Factory pipelines including parameters, triggers, dependencies, mappings, failure/retry paths, incremental loads, and watermark logic
• Test Delta Lake behavior including MERGE/upsert, SCD Type 1/2, schema evolution, deduplication, and late-arriving data
• Integrate automated tests into CI/CD pipelines using Azure DevOps, GitHub Actions, or equivalent
• Partner with the Data Quality Engineer to embed DQX/data-quality checks into pipelines
• Report defects, reconciliation results, test metrics, and go/no-go readiness to project leadership
• Support UAT, regression testing, production validation, defect resolution, and migration cutover activities
• Ensure zero critical data defects are introduced into production following each migration wave

Skills

Data QA / ETL Testing

SQL / T-SQL / Spark SQL

PySpark

Data Migration Testing

Source-to-Target Reconciliation

Data Validation / Quality Testing

Test Automation / pytest

CI/CD / Azure DevOps or GitHub Actions

Quality Gates / Test Strategy

Tools

Databricks

Delta Lake

Azure Data Factory (ADF)

Databricks Notebooks / Jobs

Unity Catalog

Azure DevOps

GitHub Actions

Job description

Job Title: Senior Data QA / Test Engineer

Duration: Contractual / Fulltime Role (W2)

Note:

We are looking for a Senior Data QA / Test Engineer with strong experience in data/ETL testing, SQL Server, Databricks, Azure Data Factory, PySpark, and data migration testing. The ideal candidate will own the testing strategy for migrating a SQL Server data platform to Databricks on Azure, validate data parity between legacy and modern platforms, establish quality gates, and ensure end-to-end validation of ADF pipelines and Delta Lake workloads.

Key Skills (50%)

• Strong experience with Data QA / ETL Testing / Data Validation
• Strong SQL experience including T-SQL, Spark SQL, joins, window functions, aggregates, and data reconciliation
• Hands-on experience with Databricks, Delta Lake, notebooks, Jobs/Workflows, and Unity Catalog
• Hands-on experience with Azure Data Factory (ADF) including pipelines, triggers, datasets, monitoring, and debugging
• Strong experience with PySpark for test automation and reconciliation frameworks
• Experience with source-to-target reconciliation, row counts, checksums/hashes, aggregates, and row-level data comparisons
• Experience testing data transformations, migrations, schema changes, incremental loads, and business logic
• Experience integrating automated tests into CI/CD using Azure DevOps or GitHub Actions

Experience (30%)

• 5+ years of experience in Data / ETL Testing
• Experience testing at least one data platform migration
• Strong experience validating SQL Server data platforms and modern cloud data platforms
• Experience converting SQL business logic, stored procedures, and views into test cases
• Experience validating PySpark / Spark SQL outputs against legacy SQL results
• Experience defining quality gates, entry/exit criteria, test metrics, and release readiness
• Experience testing ADF pipelines, dependencies, parameters, triggers, retries, and incremental/watermark logic
• Experience with Delta Lake MERGE/upsert, SCD Type 1/2, schema evolution, deduplication, and late-arriving data
• Experience with UAT, regression testing, defect management, and production validation

Other Skills (20%)

• Strong analytical, troubleshooting, and problem-solving skills
• Experience reporting defects, reconciliation results, test metrics, and go/no-go readiness
• Strong communication and collaboration with Data Engineers, Data Quality Engineers, Developers, and Project Leadership
• Experience working in Agile/Scrum environments
• Strong documentation and test strategy development skills
• Experience with pytest or similar test automation frameworks
• Knowledge of Terraform, Databricks Asset Bundles, and Azure DevOps
• Knowledge of Databricks Labs DQX, Great Expectations, Soda, or Lakeflow/DLT expectations is an asset
• Experience with Spark performance testing including partitioning, file sizing, and cluster sizing is an asset
• Knowledge of SSIS, SSRS, or Power BI report validation is an asset

Must Have

• Data / ETL Testing
• SQL / T-SQL / Spark SQL
• PySpark
• Data Migration Testing
• Source-to-Target Reconciliation
• Data Validation / Quality Testing
• Test Automation / pytest
• CI/CD / Azure DevOps or GitHub Actions
• Quality Gates / Test Strategy

Responsibilities

• Define and execute the migration test strategy and test plan covering schema, data, transformation logic, performance, and regression testing
• Build automated source-to-target reconciliation for migrated tables
• Validate row counts, column-level checksums/hashes, aggregates, and row-level differences
• Convert legacy SQL business logic, stored procedures, and views into test cases
• Validate equivalent PySpark/Spark SQL outputs against legacy SQL results
• Define and enforce quality gates across code merge, Bronze/Silver/Gold promotion, UAT sign-off, and production cutover
• Validate Azure Data Factory pipelines including parameters, triggers, dependencies, mappings, failure/retry paths, incremental loads, and watermark logic
• Test Delta Lake behavior including MERGE/upsert, SCD Type 1/2, schema evolution, deduplication, and late-arriving data
• Integrate automated tests into CI/CD pipelines using Azure DevOps, GitHub Actions, or equivalent
• Partner with the Data Quality Engineer to embed DQX/data-quality checks into pipelines
• Report defects, reconciliation results, test metrics, and go/no-go readiness to project leadership
• Support UAT, regression testing, production validation, defect resolution, and migration cutover activities
• Ensure zero critical data defects are introduced into production following each migration wave