Data Quality Engineer: Databricks, ETL & Validation Pro

Ameri-partners; North Carolina; On-site; USD 90,000 - 130,000; Full time

Job summary

Ameri Partners is seeking an experienced Data QA Engineer to ensure data accuracy, completeness, and integrity across complex data pipelines. You will work with Databricks, ETL pipelines, and automated validation scripts to validate source-to-target mappings and business rules.

The role requires hands-on testing with Python, PySpark, SQL, and data validation automation. You will troubleshoot discrepancies, collaborate with Data Engineers and stakeholders, and help mature automated testing

Qualifications

• Experience with Databricks and ETL/Data Pipeline Testing.
• Proficiency in SQL for data validation and reconciliation.
• Experience with Python and PySpark for automated data testing.
• Understanding of ETL/ELT processes, data transformations, and source-to-target mappings.
• Experience developing and executing automated data quality validation scripts.
• Ability to troubleshoot data discrepancies, investigate pipeline failures, and document defects.
• Strong analytical, problem-solving, and communication skills.

Responsibilities

• Perform end-to-end ETL testing and validate data pipelines within Databricks.
• Develop and execute automated data validation scripts using Python, PySpark, and SQL.
• Validate source-to-target mappings, data transformations, and business rules.
• Perform data reconciliation to identify missing records, duplicates, inconsistencies, and data discrepancies.
• Design and execute data quality checks to ensure accuracy, completeness, and consistency.
• Investigate data pipeline failures, troubleshoot defects, and support root cause analysis.
• Develop and maintain test cases, automated test scripts, and validation documentation.
• Collaborate with Data Engineers, Developers, and business stakeholders to resolve data quality issues.
• Support regression testing and continuous improvement of automated data testing processes.

Skills

Data QA Engineer

ETL Testing

SQL

Python/PySpark

Data Validation Automation

Tools

Databricks

Python

PySpark

SQL

Job description

Ameri Partners is seeking an experienced Data QA Engineer to ensure data accuracy, completeness, and integrity across complex data pipelines. You will work with Databricks, ETL pipelines, and automated validation scripts to validate source-to-target mappings and business rules.

The role requires hands-on testing with Python, PySpark, SQL, and data validation automation. You will troubleshoot discrepancies, collaborate with Data Engineers and stakeholders, and help mature automated testing