ETL Tester

BuzzClan; Plano (TX); On-site; USD 120,000 - 170,000; Full time

Job summary

BuzzClan is migrating its SQL Server data platform to Databricks on Azure. You will own the testing strategy, validate parity between legacy SQL and Databricks, and define quality gates for releases across the CI/CD pipeline.

You will build automated reconciliation, convert legacy logic to PySpark test cases, and validate ADF pipelines, Delta Lake behaviour, and data quality checks. This role requires strong SQL skills and Databricks experience in a hands-on, collaborative environment.

Qualifications

• 5+ years in data/ETL testing.
• Strong SQL (T-SQL and Spark SQL).
• Hands-on Databricks: notebooks, Jobs/Workflows, Delta Lake, Unity Catalog basics.
• Hands-on Azure Data Factory: pipelines, linked services, datasets, triggers, monitoring and debugging runs.
• PySpark for building reusable test and reconciliation frameworks.
• Experience defining quality gates, entry/exit criteria and test metrics.
• Test automation with pytest or similar, integrated into CI/CD.

Responsibilities

• Reconciliation framework running automatically for all migrated tables.
• Quality gates documented, agreed and enforced in the release pipeline.
• Zero critical data defects found in production after each migration wave.

Skills

ETL testing

SQL

Quality gates

CI/CD

Data platform migration

pytest

Test automation

Tools

Databricks

Azure Data Factory

PySpark

Delta Lake

Unity Catalog

CI/CD tooling

Job description

Job Type- Contractual//Fulltime

Role Context:

We are migrating our SQL Server data platform (stored procedures, SSIS/SQL jobs, reporting tables) to Databricks on Azure, orchestrated by Azure Data Factory. You will own the testing strategy for this migration: proving data parity between legacy SQL and Databricks, defining the quality gates each release must pass, and validating ADF pipelines end to end.

Key responsibilities:

Define the migration test strategy and plan covering schema, data, transformation logic, performance and regression testing.

Build automated source-to-target reconciliation: row counts, column-level checksums/hashes, aggregates, and full row-level diffs for critical tables.

Convert legacy SQL business logic (stored procedures, views) into test cases and validate the equivalent PySpark/Spark SQL output.

Define and enforce quality gates across the lifecycle: code merge, Bronze/Silver/Gold layer promotion, UAT sign-off and production cutover.

Validate ADF pipelines: parameters, triggers, dependencies, copy activity mappings, failure/retry paths, incremental loads and watermark logic.

Test Delta Lake behaviour: MERGE/upsert, SCD Type 1/2, schema evolution, deduplication and late-arriving data.

Integrate tests into CI/CD (Azure DevOps or GitHub Actions) with Databricks Asset Bundles or equivalent.

Report defects, reconciliation results and go/no-go readiness to project leadership.

Partner with the Data Quality Engineer to embed DQX checks into pipelines.

Must-have skills :

5+ years in data/ETL testing, including at least one data platform migration.

Strong SQL (T-SQL and Spark SQL): joins, window functions, EXCEPT/MINUS-based diffs, aggregates.

Hands-on Databricks: notebooks, Jobs/Workflows, Delta Lake, Unity Catalog basics.

Hands-on Azure Data Factory: pipelines, linked services, datasets, triggers, monitoring and debugging runs.

PySpark for building reusable test and reconciliation frameworks.

Experience defining quality gates, entry/exit criteria and test metrics.

Test automation with pytest or similar, integrated into CI/CD.

Nice-to-have skills :

Databricks Labs DQX, Great Expectations, Soda or Lakeflow (DLT) expectations.

Performance testing of Spark jobs (partitioning, file sizes, cluster sizing).

Knowledge of SSIS, SSRS or Power BI report validation.

Responsibilities:

Reconciliation framework running automatically for all migrated tables.

Quality gates documented, agreed and enforced in the release pipeline.

Zero critical data defects found in production after each migration wave.