DATA ENGINEER - AZURE DATABRICKS & PYSPARK

Duncan & Ross

Dubai

On-site

AED 240,000 - 360,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Duncan & Ross is hiring a Data Engineer for a leading client in Dubai. You will design, build and run scalable data pipelines and data warehouse solutions on Azure and Databricks, turning raw data into trusted analytics-ready data.

3-5+ years of hands-on data engineering experience is required. You will work with Python/PySpark, SQL, and cloud services to deliver robust data products, while ensuring data quality and governance across heterogeneous sources.

Qualifications

  • 3-5+ years hands-on data engineering experience.
  • Data pipelines: ETL/ELT pipelines for batch/incremental loads.
  • SQL & relational databases: advanced SQL with SQL Server/Azure SQL/PostgreSQL/Oracle.
  • Big data: hands-on Apache Spark, performance tuning for large datasets.
  • Cloud: strong Azure (Data Lake Storage, Data Factory, Synapse) & Databricks experience.
  • Data warehousing/lakehouse design on Azure & Databricks (dimensional modelling; Delta Lake).
  • Programming: strong Python and PySpark skills.
  • Orchestration: Azure Data Factory, Databricks Workflows, Airflow or similar.
  • Data quality, lineage, metadata, access controls; governance.

Responsibilities

  • Design, build and maintain ETL/ELT pipelines ingesting data from multiple sources into Azure and Databricks.
  • Develop data transformations in Python and PySpark, optimising Spark jobs for performance and cost.
  • Write efficient SQL for data modelling and analysis across databases.
  • Build and manage data warehouses and lakehouses on Azure and Databricks, including Delta Lake architectures.
  • Orchestrate data workflows with ADF, Databricks Workflows or Airflow, with monitoring and alerts.
  • Implement data quality checks, validation rules, and reconciliation processes.
  • Apply governance with cataloguing, lineage, access control and security (Unity Catalog / Purview).
  • Collaborate with data analysts/scientists and stakeholders to deliver data products.
  • Troubleshoot failures and performance issues; continuously improve reliability.
  • Document pipelines, models and processes; follow CI/CD and version control.

Skills

Data pipelines
SQL & databases
PySpark
Azure Databricks
Python
Azure Data Factory
Spark tuning
ETL/ELT
Git CI/CD
Agile
English proficiency

Education

Bachelor's degree

Tools

Databricks
Spark
Azure Data Factory
Airflow
CI/CD (Azure DevOps/GitHub Actions)
Power BI
Delta Lake
Purview
Terraform

Job description

Duncan & Ross is hiring a Data Engineer for a leading client in Dubai. You'll design, build and run scalable data pipelines and data warehouse solutions on Azure and Databricks, turning raw data from across the business into trusted, analytics-ready data.

What you'll do
  • Design, build and maintain ETL/ELT pipelines that ingest, transform and load data from multiple source systems into Azure and Databricks.
  • Develop data transformations in Python and PySpark, optimising Spark jobs for performance, scalability and cost.
  • Write efficient, well-structured SQL for data modelling, transformation and analysis across relational databases.
  • Build and manage data warehouse and lakehouse solutions on Azure and Databricks, including Delta Lake and medallion (bronze, silver, gold) architectures.
  • Orchestrate and schedule data workflows using tools such as Azure Data Factory, Databricks Workflows or Apache Airflow, with monitoring and alerting.
  • Put data quality checks, validation rules and reconciliation in place so data is accurate, complete and reliable.
  • Apply data governance standards, including cataloguing, lineage, access control and security, using tools such as Unity Catalog or Microsoft Purview.
  • Work with data analysts, data scientists and business stakeholders to understand requirements and deliver data products.
  • Troubleshoot pipeline failures and performance issues, and continuously improve reliability.
  • Document pipelines, data models and processes, and follow CI/CD and version control best practices.
What you'll bring
  • 3-5+ years of hands-on data engineering experience.
  • Data pipelines: proven experience building ETL/ELT pipelines for batch and incremental loads.
  • SQL and relational databases: advanced SQL and strong experience with databases such as SQL Server, Azure SQL, PostgreSQL or Oracle.
  • Big data: hands-on Apache Spark experience, including performance tuning and handling large datasets.
  • Cloud platforms: strong experience with Microsoft Azure (Data Lake Storage, Data Factory, Synapse) and Databricks.
  • Data warehousing: experience designing data warehouses or lakehouses on Azure and Databricks, including dimensional modelling (star and snowflake schemas) and Delta Lake.
  • Programming: strong Python and PySpark skills.
  • Orchestration: experience with Azure Data Factory, Databricks Workflows, Airflow or similar.
  • Data quality and governance: experience implementing data quality frameworks, data lineage, metadata management and access controls.
  • Familiarity with Git, CI/CD pipelines (Azure DevOps or GitHub Actions) and Agile delivery.
  • Bachelor's degree in computer science, Engineering, Information Systems or a related field.
  • Fluent written and spoken English.
Nice to have
  • Databricks certification (Data Engineer Associate or Professional) or Microsoft Azure Data Engineer Associate (DP-203).
  • Experience with streaming data (Kafka, Event Hubs, Spark Structured Streaming).
  • Experience with Unity Catalog, Microsoft Purview or Microsoft Fabric.
  • Infrastructure as code (Terraform or Bicep).
  • Exposure to BI tools such as Power BI.
  • Previous experience in the UAE or wider GCC.
Vertical:

Technology

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

DATA ENGINEER - AZURE DATABRICKS & PYSPARK
DATA ENGINEER - AZURE DATABRICKS & PYSPARK

Duncan & Ross Consulting • Dubai

On-site
AED 250,000 - 380,000
Azure Databricks Data Engineer - PySpark Expert
Azure Databricks Data Engineer - PySpark Expert

Duncan & Ross • Dubai

On-site
AED 240,000 - 360,000
Azure Databricks Data Engineer (PySpark)
Azure Databricks Data Engineer (PySpark)

Duncan & Ross Consulting • Dubai

On-site
AED 250,000 - 380,000
Data Engineer
Data Engineer

Datamatics Technologies LLC • Abu Dhabi

On-site
AED 240,000 - 420,000
Data Engineer
Data Engineer

Datamatics Global Services Ltd • Dubai

On-site
AED 240,000 - 360,000
Data Engineer
Data Engineer

Datamatics Technologies LLC • Dubai

On-site
AED 190,000 - 290,000
Data Engineer
Data Engineer

Datamatics Global Services Ltd • Abu Dhabi

On-site
AED 201,000 - 357,000
Data Engineer
Data Engineer

Acesoft Labs (India) Pvt Ltd • Dubai

Hybrid
AED 200,000 - 350,000
Data Engineer
Data Engineer

Datamatics Technologies • Sharjah

On-site
AED 240,000 - 360,000
Data Engineer
Data Engineer

Client of AIQU • Abu Dhabi

On-site
AED 350,000 - 550,000
Top locations