Azure Databricks Data Engineer (PySpark)

Duncan & Ross Consulting; Dubai; On-site; AED 250,000 - 380,000; Full time

Job summary

Duncan & Ross Consulting in Dubai is seeking a Data Engineer to design, build, and run scalable data pipelines and data warehouse solutions on Azure and Databricks. You will turn raw data from across the business into trusted, analytics-ready data.

You will collaborate with data analysts, data scientists and business stakeholders, implement data governance and quality checks, and ensure reliable, secure data products for analytics across the organization.

Qualifications

• 3-5+ years hands-on data engineering experience.
• Experience building ETL/ELT pipelines for batch and incremental loads.
• Advanced SQL and strong experience with databases such as SQL Server, Azure SQL, PostgreSQL or Oracle.
• Hands-on Apache Spark experience, including performance tuning.
• Strong Microsoft Azure (Data Lake Storage, Data Factory, Synapse) and Databricks experience.
• Experience designing data warehouses or lakehouses on Azure and Databricks, including dimensional modelling and Delta Lake.

Responsibilities

• Design, build and maintain ETL/ELT pipelines ingesting data from multiple sources into Azure and Databricks.
• Develop data transformations in Python and PySpark for performance and scalability.
• Write efficient SQL for data modelling and analysis across relational databases.
• Build and manage data warehouse/lakehouse solutions on Azure and Databricks with Delta Lake.
• Orchestrate data workflows using Azure Data Factory, Databricks Workflows or Airflow, with monitoring.
• Implement data quality checks, validation rules, and data reconciliation.
• Follow data governance standards including cataloguing, lineage, access control and security.
• Collaborate with data analysts, data scientists and business stakeholders to deliver data products.
• Troubleshoot pipelines and improve reliability; document pipelines and CI/CD practices.

Job description

Duncan & Ross Consulting in Dubai is seeking a Data Engineer to design, build, and run scalable data pipelines and data warehouse solutions on Azure and Databricks. You will turn raw data from across the business into trusted, analytics-ready data.

You will collaborate with data analysts, data scientists and business stakeholders, implement data governance and quality checks, and ensure reliable, secure data products for analytics across the organization.