Data Quality Tester for Azure Data Factory & Databricks

Compunnel, Inc.; Southwestern Ontario; On-site; CAD 70,000 - 90,000; Full time

Job summary

A data solutions company in Southwestern Ontario is looking for a Data Quality Tester to validate data pipelines and ensure data integrity across Azure and Databricks environments. The role involves developing test strategies and automation scripts while collaborating with engineering teams to maintain high data quality standards.

Qualifications

• Strong knowledge of Azure Data Factory and Databricks.
• Proficiency in SQL and scripting languages (Python preferred).
• Experience with data profiling, reconciliation, and schema validation.
• Familiarity with DevOps tools like Azure DevOps or GitHub Actions.
• Understanding of SCD Type 2 and transformation logic.

Responsibilities

• Develop and execute test strategies and automation scripts for data pipelines.
• Validate ETL/ELT pipelines built using Azure Data Factory and Databricks.
• Perform data validation, reconciliation, and comparative analysis between source and target systems.
• Monitor data quality across Delta tables and data warehouses.
• Integrate data testing into CI/CD workflows.
• Document defects and collaborate with engineering teams for resolution.

Skills

Strong knowledge of Azure Data Factory

Proficiency in SQL

Scripting languages (Python preferred)

Experience with data profiling

Familiarity with DevOps tools

Job description

We are looking for a Data Quality Tester to validate data pipelines and ensure data integrity across Azure and Databricks environments.

This role involves developing test strategies, automation scripts, and collaborating with engineering teams to maintain high data quality standards.

Key Responsibilities

• Develop and execute test strategies and automation scripts for data pipelines.
• Validate ETL/ELT pipelines built using Azure Data Factory and Databricks.
• Perform data validation, reconciliation, and comparative analysis between source and target systems.
• Monitor data quality across Delta tables and data warehouses.
• Integrate data testing into CI/CD workflows.
• Document defects and collaborate with engineering teams for resolution.

Required Qualifications

• Strong knowledge of Azure Data Factory and Databricks.
• Proficiency in SQL and scripting languages (Python preferred).
• Experience with data profiling, reconciliation, and schema validation.
• Familiarity with DevOps tools like Azure DevOps or GitHub Actions.
• Understanding of SCD Type 2 and transformation logic.

Email ID * This field is required Please enter valid emailId.

Cell phone * This field is required Please enter valid cell phone.

First Name * This field is required Please enter valid first name.

Last Name * This field is required Please enter valid last name.