Azure Databricks Tech Lead - Lakehouse & Platform Architect

Birbal AI

Abu Dhabi

On-site

AED 260,000 - 380,000

Full time

9 days ago

Application generator

Stand out for this role — generate a tailored resume and cover letter in about a minute.

Get past ATS filters

Job summary

Birbal AI seeks an experienced Technical Lead – Azure Databricks to drive the delivery of a modern data platform on Azure. You will own platform architecture, ingestion pipelines and governance, leading discovery, design and implementation with a strong client-facing focus.

You will manage cross-functional Azure teams, set up CI/CD with Azure DevOps, guide two-week sprints from planning to demonstrations, and ensure security, cost controls and optimization. UAE-based or relocate to Abu Dhabi.

Qualifications

10+ years in Data Engineering/Data Platform delivery.
4+ years hands-on Azure Databricks.
Led at least one complete Azure Databricks Lakehouse implementation end-to-end.
Strong PySpark, SQL, Delta Lake and Unity Catalog.
On-premise DB-to-Azure integration experience.
Azure DevOps CI/CD and Databricks deployment experience.
Experience leading remote/offshore Agile teams.
Fluent English and UAE-based or willing to relocate to Abu Dhabi.
Strong client-facing communication skills.

Responsibilities

Lead discovery phase covering source-system profiling, ingestion requirements and delivery planning.
Own Azure Databricks platform architecture, configuration and delivery.
Design environments, Unity Catalog governance, security, access controls, monitoring and cost controls.
Lead development of metadata-driven ingestion frameworks and pipelines.
Manage integration of on-premise SQL Server and Oracle Fusion HCM/Finance data.
Establish CI/CD using Azure DevOps.
Oversee semantic-layer development and Databricks Genie Agents.
Lead two-week Agile sprints and client demonstrations.
Conduct code/design reviews and knowledge transfer.