Data Quality Engineer (Databricks)

Salt Digital Recruitment; Dubai; On-site; AED 223,200 - 334,800; Full time

Job summary

Salt Digital Recruitment is seeking an expert in Data Quality Management based in Dubai, UAE. The role involves developing Delta Lake metrics, setting up automated quality monitoring systems, and performing root cause analysis to enhance data quality.

The ideal candidate should have strong Databricks engineering skills, proficiency in PySpark, and experience in data governance and compliance. This role offers opportunities to work on pioneering data quality projects.

Qualifications

• Strong experience designing and implementing enterprise Data Quality frameworks.
• Advanced Databricks engineering experience.
• Strong PySpark development skills.
• Experience with Delta Lake, Unity Catalog, Databricks Workflows and Jobs, Databricks SQL.
• Knowledge of machine learning techniques for anomaly detection.

Responsibilities

• Develop Delta Lake metric aggregation structures for quality reporting.
• Configure automated quality monitoring and alerting mechanisms.
• Support AI-assisted root cause analysis and remediation.
• Collaborate with stakeholders for governance and compliance.

Skills

Data Quality Framework Design

Databricks Engineering

PySpark Development

Delta Lake

Machine Learning

Data Governance

Power BI Integration

Cloud Data Engineering

Job description

Overview

Develop Delta Lake metric aggregation structures supporting enterprise quality reporting. Calculate and publish: Data Quality Index (DQI) scores; dimension-level quality metrics; rule pass/fail rates; dataset compliance scores; SLA adherence indicators. Provide curated outputs to support Power BI quality dashboards and executive reporting.

Monitoring, Alerting and Predictive Quality Management

Configure automated quality monitoring and alerting mechanisms. Implement threshold-based notifications using Databricks SQL Alerts, Azure Monitor integrations. Develop predictive risk scoring models to identify datasets at risk of future quality degradation. Support proactive quality management and operational intervention activities.

Root Cause Analysis and Remediation Support

Apply Databricks machine learning and pattern analysis techniques to profiling and rule execution outputs. Support AI-assisted root cause analysis across established remediation categories. Identify recurring quality issues, systemic defects, and process breakdowns. Produce prioritised remediation datasets for business and operational stakeholders. Export remediation outputs to Power BI and Excel to support Remediation Tiering Matrix, Prioritisation Scoring Models, Governance reporting processes.

Governance, Compliance and Stakeholder Engagement

Collaborate with Data Modellers and Data Catalogue Specialists to ensure quality controls align with authoritative data definitions and metadata standards. Support DDA and DGE governance processes by producing required quality artefacts and evidence. Maintain documentation, version control, and audit trails for all quality assets, rules, models, and processes. Participate in quality reviews, governance forums, and stakeholder workshops.

Required Skills and Experience

• Strong experience designing and implementing enterprise Data Quality frameworks.
• Advanced Databricks engineering experience.
• Strong PySpark development skills.
• Experience with Delta Lake, Unity Catalog, Databricks Workflows and Jobs, Databricks SQL.
• Experience building scalable data validation and quality rule frameworks.
• Knowledge of machine learning techniques for anomaly detection and data quality monitoring.
• Experience using MLflow for model management and deployment.
• Strong understanding of data governance, metadata management, and data lifecycle processes.
• Experience integrating data quality metrics into reporting platforms such as Power BI.
• Knowledge of cloud-based data engineering and modern Lakehouse architectures.

Deliverables

• Configured Databricks Data Quality environment.
• Enterprise Data Quality Rule Factory.
• AI-assisted profiling notebooks and baseline assessment outputs.
• Automated data quality validation and gating processes.
• Data cleansing and remediation pipelines.
• Failed Record Register and reprocessing workflows.
• Data Quality metric aggregation tables.
• DQI reporting feeds and dashboard datasets.
• Predictive quality monitoring and alerting solutions.
• Root Cause Analysis and remediation support datasets.
• Governance, audit, and compliance artefacts supporting DDA and DGE reviews.

Salt is acting as an Employment Business in relation to this vacancy.