Data Quality Engineer - Azure/Databricks, Hybrid Cape Town

Embelo; Cape Town; Hybride; ZAR 600 000 - 900 000; Plein temps

Résumé du poste

Embelo’s Inference Group is seeking a hands-on Data Quality Assurance Engineer to join the Data and Integration team in Cape Town. You will ensure Azure and Databricks data is accurate, complete and trustworthy by profiling, validating, reconciling and monitoring data end-to-end.

You will design automated checks with SQL and PySpark, build data quality monitoring, and contribute from week one in a delivery role. Hybrid work from Woodstock, with office presence required several days a week.

Qualifications

• Hands-on data quality, data testing or data engineering experience.
• Experience in Microsoft Azure data environments.
• Databricks experience and advanced SQL skills.

Responsabilités

• Profile source and target datasets for completeness and anomalies.
• Design and implement data validation rules and automated checks using SQL and PySpark.
• Reconcile data between source systems, staging layers and target models.
• Build and maintain data quality monitoring, alerting and reporting.
• Perform root cause analysis on data issues with Data Engineers to resolve them.
• Test data pipelines, transformations, and data models before and after release.
• Embed data quality tests into Azure DevOps CI/CD pipelines.
• Define and track data quality metrics and thresholds with stakeholders.
• Help define data quality standards, frameworks and best practices across client projects.
• Support master data and reference data quality initiatives.
• Guide and support less experienced team members on data quality practices.
• Work with Data Engineers, Integration Developers, Analysts and business data owners.
• Document data quality rules, findings and processes, and keep stakeholders updated on progress.

Connaissances

Azure data environments

Databricks

Advanced SQL

PySpark

Data profiling

Data reconciliation

CI/CD

Agile

Independent work

Formation

Degree in Computer Science / Information Systems

Outils

Azure DevOps

dbt

Python

Description du poste

Embelo’s Inference Group is seeking a hands-on Data Quality Assurance Engineer to join the Data and Integration team in Cape Town. You will ensure Azure and Databricks data is accurate, complete and trustworthy by profiling, validating, reconciling and monitoring data end-to-end.

You will design automated checks with SQL and PySpark, build data quality monitoring, and contribute from week one in a delivery role. Hybrid work from Woodstock, with office presence required several days a week.