Data Engineer: Azure Databricks, PySpark & ML Prep

Duncan & Ross; Abu Dhabi; On-site; AED 240,000 - 360,000; Full time

Job summary

Duncan & Ross in Abu Dhabi seeks a Senior Data Engineer to design and maintain scalable data pipelines for ingestion, transformation and storage.

Collaborate with data scientists and analysts to enable data access while upholding data governance, privacy and security best practices. Expert in SQL, Python and Azure Databricks with PySpark and Data Factory or Synapse is required; 5+ years of experience in data engineering.

Qualifications

• Minimum 5 years in data engineering or data platform role.
• Experience with Azure Databricks, Data Lake Storage, PySpark, and Jupyter Notebooks.
• Proficiency in SQL, Python, and distributed data processing frameworks.
• Experience with Azure Data Factory, Azure Synapse, or similar ETL/ELT tools.
• Understanding of data governance, privacy, and security best practices.
• Familiarity with supporting ML/AI workflows including data preparation for machine learning pipelines.
• Excellent problem-solving and communication skills.

Responsibilities

• Design, build, and maintain data pipelines for ingestion, transformation, and storage.
• Ensure data integrity, quality, and security throughout the data lifecycle.
• Optimize data processing workflows for performance and scalability.
• Collaborate with data scientists and analysts to facilitate data accessibility.

Skills

5+ years data engineering

SQL

Python

Distributed processing

Data governance

ML data prep

Problem solving

Communication

Tools

Azure Databricks

Data Lake Storage

PySpark

Jupyter Notebooks

Azure Data Factory

Azure Synapse

Job description

Duncan & Ross in Abu Dhabi seeks a Senior Data Engineer to design and maintain scalable data pipelines for ingestion, transformation and storage.

Collaborate with data scientists and analysts to enable data access while upholding data governance, privacy and security best practices. Expert in SQL, Python and Azure Databricks with PySpark and Data Factory or Synapse is required; 5+ years of experience in data engineering.