Senior Data QA Engineer: ETL/Databricks Automation

Beacon Hill Staffing Group; Wayne (PA); Presencial; USD 115.000 - 125.000; Jornada completa

Descripción de la vacante

Beacon Hill Staffing Group seeks a Senior Data QA Engineer to lead end-to-end QA of data pipelines in Wayne, PA. You will validate mappings, transformations, and business rules, and write complex SQL for data validation across ETL processes.

Ideal candidates have 8+ years in data QA, hands-on ETL testing, and experience with Azure Databricks, Delta Lake, DLT, ADF, Python/PySpark, and CI/CD to drive QA automation and data quality initiatives.

Formación

• 8+ years of Data QA/data testing experience.
• Hands-on ETL/data pipeline testing.
• Advanced, hands-on SQL for data validation.
• Strong ETL/ELT testing and source-to-target reconciliation.
• Data transformation and business-rule validation.
• Experience with Azure Databricks, Delta Lake, DLT and Workflows.
• Azure Data Factory pipelines and data validation.
• Python and/or PySpark for QA automation.
• Data quality and pipeline regression automation.
• Azure DevOps and CI/CD.
• Strong data troubleshooting and root-cause analysis.
• Hands-on contributor who can drive QA strategy.

Responsabilidades

• Perform end-to-end QA of data and ETL/ELT pipelines.
• Validate source-to-target mappings, transformations, and business rules.
• Write complex SQL for data validation and reconciliation.
• Design test plans, test cases, and automated data tests.
• Validate data completeness, accuracy, integrity, and consistency.
• Build QA automation for Databricks, DLT, and ADF.
• Implement schema, constraint, and reconciliation checks.
• Automate ADF pre/post-load and schema drift validation.
• Build reusable QA frameworks using SQL, Python, and PySpark.
• Perform regression, volume, load, and pipeline testing.
• Troubleshoot data issues across source, transformation, and target layers.
• Partner with data engineers, architects, governance, and business teams.
• Document test results, defects, traceability, and audit evidence.

Conocimientos

Data QA

SQL mastery

ETL testing

Data validation

QA automation

Python/PySpark

CI/CD

Root-cause analysis

Hands-on QA

Herramientas

Azure Databricks

Delta Lake

DLT

Workflows

Azure Data Factory

Azure DevOps

Descripción del empleo

Beacon Hill Staffing Group seeks a Senior Data QA Engineer to lead end-to-end QA of data pipelines in Wayne, PA. You will validate mappings, transformations, and business rules, and write complex SQL for data validation across ETL processes.

Ideal candidates have 8+ years in data QA, hands-on ETL testing, and experience with Azure Databricks, Delta Lake, DLT, ADF, Python/PySpark, and CI/CD to drive QA automation and data quality initiatives.