Databricks Data Quality & Remediation Architect Lead

iO Associates; Abu Dhabi; Hybrid; AED 350,000 - 550,000; Part time

Job summary

iO Associates is partnering with a leading UAE organisation to recruit a Databricks Data Quality & Remediation Architect / Lead for a hybrid role in Abu Dhabi. This senior, hands-on position combines Databricks architecture, data engineering and technical leadership on a major data platform delivery.

You will design scalable data quality capabilities, define Bronze-Silver-Gold layers, work with governance teams and lead a development team to deliver measurable quality KPIs and remediation

Qualifications

• 5+ years' experience in data engineering, Databricks or related roles.
• Ideally 2+ years in a Lead or Architect position.
• Strong hands on Databricks experience.
• Excellent Python and PySpark skills.

Responsibilities

• Design and implement a scalable Data Quality & Remediation Engine on Databricks.
• Define architecture across data profiling, quality rules, exception management, root-cause analysis and remediation.
• Build a metadata-driven data-quality framework.
• Define Bronze, Silver and Gold processing layers.
• Work with Data Governance teams and Data Stewards on Critical Data Elements and data-quality processes.
• Support Microsoft Purview integration, governance, lineage and auditability.
• Lead the development team and provide hands on technical direction.
• Own data-quality reporting, including quality KPIs, exceptions and remediation activity.

Skills

Databricks

Python

PySpark

Azure Databricks

Data quality

Leadership

Stakeholder management

Delta Lake

Databricks SQL

Tools

Azure Data Factory

ADLS Gen2

Azure DevOps

Purview

Job description

Databricks Data Quality & Remediation Architect / Lead

Location: Hybrid - UAE | Abu Dhabi preferred
Contract: 6 months initially, with a strong likelihood of extension
Rate: Competitive, depending on experience
Start: ASAP

Our client is a leading organisation in the UAE delivering a major enterprise data transformation programme, and they are looking for a Databricks Data Quality & Remediation Architect / Lead Developer to help design and lead the delivery of a new enterprise-scale data quality platform.

This is a senior, hands-on role combining Databricks architecture, data engineering and technical leadership.

What You'll Do

• Design and implement a scalable Data Quality & Remediation Engine on Databricks.
• Define architecture across data profiling, quality rules, exception management, root-cause analysis and remediation.
• Build a metadata-driven data-quality framework.
• Define Bronze, Silver and Gold processing layers.
• Work with Data Governance teams and Data Stewards on Critical Data Elements and data-quality processes.
• Support Microsoft Purview integration, governance, lineage and auditability.
• Lead the development team and provide hands on technical direction.
• Own data-quality reporting, including quality KPIs, exceptions and remediation activity.

What We're Looking For

• 5+ years' experience in data engineering, Databricks or related roles.
• Ideally 2+ years in a Lead or Architect position.
• Strong hands on Databricks experience.
• Excellent Python and PySpark skills.
• Strong knowledge of Databricks Lakehouse, Delta Lake, Unity Catalog, Workflows, Lakeflow and Databricks SQL.
• Experience with Azure Databricks and services such as Data Factory, ADLS Gen2 and Azure DevOps.
• Experience with data quality, governance or remediation solutions.
• Strong technical leadership and stakeholder-management skills.

Desirable

• Microsoft Purview.
• SAP S/4HANA, SAP BW or Oracle Fusion.
• MLflow, Mosaic AI or Databricks Asset Bundles.
• Enterprise data governance and lineage.

Why Apply?

You'll have the opportunity to shape the architecture of a major enterprise data platform, lead delivery and work on complex data-quality challenges at scale.

The programme is expected to continue beyond the initial six-month contract, offering strong potential for long-term engagement.

If you're an experienced Databricks Lead or Architect looking for a technically challenging project where you can make a real impact, we'd love to hear from you.