Senior Data QA Engineer

Beacon Hill Staffing Group; Wayne (PA); Vor Ort; USD 115.000 - 125.000; Vollzeit

Zusammenfassung

Beacon Hill Staffing Group seeks a Senior Data QA Engineer to lead end-to-end QA of data pipelines in Wayne, PA. You will validate mappings, transformations, and business rules, and write complex SQL for data validation across ETL processes.

Ideal candidates have 8+ years in data QA, hands-on ETL testing, and experience with Azure Databricks, Delta Lake, DLT, ADF, Python/PySpark, and CI/CD to drive QA automation and data quality initiatives.

Qualifikationen

• 8+ years of Data QA/data testing experience.
• Hands-on ETL/data pipeline testing.
• Advanced, hands-on SQL for data validation.
• Strong ETL/ELT testing and source-to-target reconciliation.
• Data transformation and business-rule validation.
• Experience with Azure Databricks, Delta Lake, DLT and Workflows.
• Azure Data Factory pipelines and data validation.
• Python and/or PySpark for QA automation.
• Data quality and pipeline regression automation.
• Azure DevOps and CI/CD.
• Strong data troubleshooting and root-cause analysis.
• Hands-on contributor who can drive QA strategy.

Aufgaben

• Perform end-to-end QA of data and ETL/ELT pipelines.
• Validate source-to-target mappings, transformations, and business rules.
• Write complex SQL for data validation and reconciliation.
• Design test plans, test cases, and automated data tests.
• Validate data completeness, accuracy, integrity, and consistency.
• Build QA automation for Databricks, DLT, and ADF.
• Implement schema, constraint, and reconciliation checks.
• Automate ADF pre/post-load and schema drift validation.
• Build reusable QA frameworks using SQL, Python, and PySpark.
• Perform regression, volume, load, and pipeline testing.
• Troubleshoot data issues across source, transformation, and target layers.
• Partner with data engineers, architects, governance, and business teams.
• Document test results, defects, traceability, and audit evidence.

Kenntnisse

Data QA

SQL mastery

ETL testing

Data validation

QA automation

Python/PySpark

CI/CD

Root-cause analysis

Hands-on QA

Tools

Azure Databricks

Delta Lake

DLT

Workflows

Azure Data Factory

Azure DevOps

Jobbeschreibung

Senior Data QA Engineer (T1480674-PHI) Wayne, Pennsylvania

Salary: USD115000 - USD125000 per year

Location: 1 day/ month on site in Wayne, PA

Position Description

• Perform end-to-end QA of data and ETL/ELT pipelines
• Validate source-to-target mappings, transformations, and business rules
• Write complex SQL for data validation and reconciliation
• Design test plans, test cases, and automated data tests
• Validate data completeness, accuracy, integrity, and consistency
• Build QA automation for Databricks, DLT, and ADF
• Implement schema, constraint, and reconciliation checks
• Automate ADF pre/post-load and schema drift validation
• Build reusable QA frameworks using SQL, Python, and PySpark
• Perform regression, volume, load, and pipeline testing
• Troubleshoot data issues across source, transformation, and target layers
• Partner with data engineers, architects, governance, and business teams
• Document test results, defects, traceability, and audit evidence

Required Skills

• 8+ years of Data QA/data testing experience
• True Data QA background with hands-on ETL/data pipeline testing
• Advanced, hands-on SQL for complex data validation
• Strong ETL/ELT testing and source-to-target reconciliation
• Data transformation and business-rule validation
• Azure Databricks, Delta Lake, DLT, and Workflows
• Azure Data Factory pipelines and data validation
• Python and/or PySpark for QA automation
• Data quality and pipeline regression automation
• Azure DevOps and CI/CD
• Strong data troubleshooting and root-cause analysis
• Ability to work hands-on while helping drive QA strategy