Senior Databricks Data Platform Engineer

Epergne Solutions; Abu Dhabi; On-site; AED 279,000 - 390,600; Full time

Job summary

A technology solutions provider in Abu Dhabi is seeking a Databricks Data Platform Engineer to design and build scalable data pipelines on Azure Databricks. The ideal candidate will have 5+ years of hands-on experience with Azure Databricks, Spark, and PySpark, along with expertise in Delta Lake and data governance frameworks. Responsibilities include developing batch and streaming pipelines, optimizing Spark workloads, and ensuring compliance with governance standards. This role offers a dynamic environment working on enterprise data initiatives.

Qualifications

• 5+ years of experience with Azure Databricks, Spark, and PySpark.
• Hands-on knowledge of Delta Lake and Structured Streaming.
• Experience in CI/CD and DevOps in data platform environments.

Responsibilities

• Develop batch pipelines using Delta Lake with best practices.
• Implement incremental ingestion with Autoloader techniques.
• Build and optimize Structured Streaming pipelines.

Skills

Azure Databricks

Spark

PySpark

Delta Lake

Autoloader

Structured Streaming

Lakeflow

CI/CD

DevOps

Job description

A technology solutions provider in Abu Dhabi is seeking a Databricks Data Platform Engineer to design and build scalable data pipelines on Azure Databricks. The ideal candidate will have 5+ years of hands-on experience with Azure Databricks, Spark, and PySpark, along with expertise in Delta Lake and data governance frameworks. Responsibilities include developing batch and streaming pipelines, optimizing Spark workloads, and ensuring compliance with governance standards. This role offers a dynamic environment working on enterprise data initiatives.