Databricks Data Platform Engineer

Epergne Solutions; Abu Dhabi; On-site; AED 279,000 - 390,600; Full time

Job summary

A technology solutions provider in Abu Dhabi is seeking a Databricks Data Platform Engineer to design and build scalable data pipelines on Azure Databricks. The ideal candidate will have 5+ years of hands-on experience with Azure Databricks, Spark, and PySpark, along with expertise in Delta Lake and data governance frameworks. Responsibilities include developing batch and streaming pipelines, optimizing Spark workloads, and ensuring compliance with governance standards. This role offers a dynamic environment working on enterprise data initiatives.

Qualifications

• 5+ years of experience with Azure Databricks, Spark, and PySpark.
• Hands-on knowledge of Delta Lake and Structured Streaming.
• Experience in CI/CD and DevOps in data platform environments.

Responsibilities

• Develop batch pipelines using Delta Lake with best practices.
• Implement incremental ingestion with Autoloader techniques.
• Build and optimize Structured Streaming pipelines.

Skills

Azure Databricks

Spark

PySpark

Delta Lake

Autoloader

Structured Streaming

Lakeflow

CI/CD

DevOps

Job description

Overview

Job Role: Databricks Data Platform Engineer

Job Location: Abu Dhabi, UAE

Experience: 5+ Years

Role Summary: Design and build scalable, high-performance, and governed data pipelines on Azure Databricks to support analytics, reporting, and enterprise data intelligence initiatives.

Responsibilities

• Key Responsibilities – Develop batch pipelines using Delta Lake with proper table design, partitioning, optimization, and lifecycle management
• Implement incremental ingestion using Autoloader with schema evolution and checkpointing
• Build Structured Streaming pipelines with watermarking, state management, and late data handling
• Develop declarative pipelines using Lakeflow standards
• Ensure pipelines support idempotency, replayability, and safe backfills
• Optimize Spark workloads (AQE, skew handling, shuffle tuning, join optimization)
• Build curated, analytics-ready datasets for SQL analytics and downstream consumption
• Develop DBSQL views aligned with enterprise semantic standards
• Implement CI/CD deployments using Repos and asset-based packaging
• Add monitoring, logging, and operational observability to pipelines

• Secondary Responsibilities – Ensure compliance with Unity Catalog access controls and governance policies
• Embed data quality validations as per governance standards
• Support migration of legacy data workloads to Databricks including validation and performance tuning
• Collaborate with DataOps for deployment readiness and production stability

Qualifications

• Strong hands-on experience with Azure Databricks, Spark, and PySpark
• Deep knowledge of Delta Lake, Autoloader, and Structured Streaming
• Experience with Lakeflow pipelines and DBSQL development
• Expertise in Spark performance tuning and optimization techniques
• Understanding of data governance, access control, and data quality frameworks
• CI/CD and DevOps experience in data platform environments
• Experience supporting production-grade data platforms