Data Architect: AI-Driven Lakehouse & Pipelines Expert

Client of Discovered MENA; Abu Dhabi; On-site; AED 350,000 - 600,000; Full time

Job summary

Client of Discovered MENA in Abu Dhabi seeks an experienced Data Architect/Data Engineer to design scalable data architectures using the Databricks Lakehouse Platform for analytical and AI workloads.

You will build and optimize ETL/ELT pipelines, collaborate with AI teams to productionize ML use cases, and establish governance, security, and metadata practices. Strong Python/SQL and PySpark skills with Azure Cloud experience are essential.

Qualifications

• 7+ years of experience in Data Engineering, Data Architecture or Enterprise Data Platform roles.
• Extensive hands-on experience designing and delivering enterprise solutions using Databricks and modern lakehouse architectures.
• Proven experience supporting the delivery of AI and Machine Learning use cases into production, working closely with Data Science and AI Engineering teams.
• Strong expertise in Python, SQL and PySpark, with experience building scalable ETL/ELT pipelines using tools such as Airflow, dbt or similar.
• Strong understanding of distributed computing, data modelling, big data processing and performance optimisation.
• Experience with cloud-based data platforms, ideally Azure, and familiarity with MLOps and the data requirements of production AI systems.
• Experience across data governance, quality, security, lineage and metadata management.
• Excellent problem-solving and stakeholder management skills, with the ability to communicate technical concepts clearly.

Responsibilities

• Design and implement scalable data architectures using the Databricks Lakehouse Platform supporting both analytical and AI workloads
• Build and optimise ETL ELT pipelines for large volumes of structured and unstructured data including real-time and batch processing
• Partner with AI and Data Science teams to deliver the data capabilities required for production ML predictive analytics and Generative AI applications
• Establish best practices across data modelling governance quality security lineage and metadata management
• Optimise data platform performance scalability and cost efficiency across cloud environments
• Collaborate with engineering and business stakeholders to translate requirements into robust technical solutions and support the integration of AI into enterprise applications
• Document architecture standards and contribute to the continuous improvement of the organisation s data platforms and engineering practices

Skills

Databricks Lakehouse Platform

Python

SQL

PySpark

ETL/ELT pipelines

Cloud platforms (Azure)

Data governance

MLOps awareness

Education

Bachelor's degree in Computer Science, Engineering, Information Systems or a related discipline

Tools

Airflow

dbt

Job description

Client of Discovered MENA in Abu Dhabi seeks an experienced Data Architect/Data Engineer to design scalable data architectures using the Databricks Lakehouse Platform for analytical and AI workloads.

You will build and optimize ETL/ELT pipelines, collaborate with AI teams to productionize ML use cases, and establish governance, security, and metadata practices. Strong Python/SQL and PySpark skills with Azure Cloud experience are essential.