DATA ENGINEER - AZURE DATABRICKS & PYSPARK

Duncan & Ross Consulting; Dubai; On-site; AED 250,000 - 380,000; Full time

Job summary

Duncan & Ross Consulting in Dubai is seeking a Data Engineer to design, build, and run scalable data pipelines and data warehouse solutions on Azure and Databricks. You will turn raw data from across the business into trusted, analytics-ready data.

You will collaborate with data analysts, data scientists and business stakeholders, implement data governance and quality checks, and ensure reliable, secure data products for analytics across the organization.

Qualifications

• 3-5+ years hands-on data engineering experience.
• Experience building ETL/ELT pipelines for batch and incremental loads.
• Advanced SQL and strong experience with databases such as SQL Server, Azure SQL, PostgreSQL or Oracle.
• Hands-on Apache Spark experience, including performance tuning.
• Strong Microsoft Azure (Data Lake Storage, Data Factory, Synapse) and Databricks experience.
• Experience designing data warehouses or lakehouses on Azure and Databricks, including dimensional modelling and Delta Lake.

Responsibilities

• Design, build and maintain ETL/ELT pipelines ingesting data from multiple sources into Azure and Databricks.
• Develop data transformations in Python and PySpark for performance and scalability.
• Write efficient SQL for data modelling and analysis across relational databases.
• Build and manage data warehouse/lakehouse solutions on Azure and Databricks with Delta Lake.
• Orchestrate data workflows using Azure Data Factory, Databricks Workflows or Airflow, with monitoring.
• Implement data quality checks, validation rules, and data reconciliation.
• Follow data governance standards including cataloguing, lineage, access control and security.
• Collaborate with data analysts, data scientists and business stakeholders to deliver data products.
• Troubleshoot pipelines and improve reliability; document pipelines and CI/CD practices.

Job description

Duncan & Ross is hiring a Data Engineer for a leading client in Dubai. You'll design, build and run scalable data pipelines and data warehouse solutions on Azure and Databricks, turning raw data from across the business into trusted, analytics-ready data.

What you'll do

• Design, build and maintain ETL/ELT pipelines that ingest, transform and load data from multiple source systems into Azure and Databricks.
• Develop data transformations in Python and PySpark, optimising Spark jobs for performance, scalability and cost.
• Write efficient, well-structured SQL for data modelling, transformation and analysis across relational databases.
• Build and manage data warehouse and lakehouse solutions on Azure and Databricks, including Delta Lake and medallion (bronze, silver, gold) architectures.
• Orchestrate and schedule data workflows using tools such as Azure Data Factory, Databricks Workflows or Apache Airflow, with monitoring and alerting.
• Put data quality checks, validation rules and reconciliation in place so data is accurate, complete and reliable.
• Apply data governance standards, including cataloguing, lineage, access control and security, using tools such as Unity Catalog or Microsoft Purview.
• Work with data analysts, data scientists and business stakeholders to understand requirements and deliver data products.
• Troubleshoot pipeline failures and performance issues, and continuously improve reliability.
• Document pipelines, data models and processes, and follow CI/CD and version control best practices.

What you'll bring

• 3-5+ years of hands-on data engineering experience.
• Data pipelines: proven experience building ETL/ELT pipelines for batch and incremental loads.
• SQL and relational databases: advanced SQL and strong experience with databases such as SQL Server, Azure SQL, PostgreSQL or Oracle.
• Big data: hands-on Apache Spark experience, including performance tuning and handling large datasets.
• Cloud platforms: strong experience with Microsoft Azure (Data Lake Storage, Data Factory, Synapse) and Databricks.
• Data warehousing: experience designing data warehouses or lakehouses on Azure and Databricks, including dimensional modelling (star and snowflake schemas) and Delta Lake.
• Programming: strong Python and PySpark skills.
• Orchestration: experience with Azure Data Factory, Databricks Workflows, Airflow or similar.
• Data quality and governance: experience implementing data quality frameworks, data lineage, metadata management and access controls.
• Familiarity with Git, CI/CD pipelines (Azure DevOps or GitHub Actions) and Agile delivery.
• Bachelor's degree in computer science, Engineering, Information Systems or a related field.
• Fluent written and spoken English.

Nice to have

• Databricks certification (Data Engineer Associate or Professional) or Microsoft Azure Data Engineer Associate (DP-203).
• Experience with streaming data (Kafka, Event Hubs, Spark Structured Streaming).
• Experience with Unity Catalog, Microsoft Purview or Microsoft Fabric.
• Infrastructure as code (Terraform or Bicep).
• Exposure to BI tools such as Power BI.
• Previous experience in the UAE or wider GCC.

Vertical:

Technology