Technical Lead-Azure Databricks

Birbal AI; Abu Dhabi; On-site; AED 260,000 - 380,000; Full time

Job summary

Birbal AI seeks an experienced Technical Lead – Azure Databricks to drive the delivery of a modern data platform on Azure. You will own platform architecture, ingestion pipelines and governance, leading discovery, design and implementation with a strong client-facing focus.

You will manage cross-functional Azure teams, set up CI/CD with Azure DevOps, guide two-week sprints from planning to demonstrations, and ensure security, cost controls and optimization. UAE-based or relocate to Abu Dhabi.

Qualifications

• 10+ years in Data Engineering/Data Platform delivery.
• 4+ years hands-on Azure Databricks.
• Led at least one complete Azure Databricks Lakehouse implementation end-to-end.
• Strong PySpark, SQL, Delta Lake and Unity Catalog.
• On-premise DB-to-Azure integration experience.
• Azure DevOps CI/CD and Databricks deployment experience.
• Experience leading remote/offshore Agile teams.
• Fluent English and UAE-based or willing to relocate to Abu Dhabi.
• Strong client-facing communication skills.

Responsibilities

• Lead discovery phase covering source-system profiling, ingestion requirements and delivery planning.
• Own Azure Databricks platform architecture, configuration and delivery.
• Design environments, Unity Catalog governance, security, access controls, monitoring and cost controls.
• Lead development of metadata-driven ingestion frameworks and pipelines.
• Manage integration of on-premise SQL Server and Oracle Fusion HCM/Finance data.
• Establish CI/CD using Azure DevOps.
• Oversee semantic-layer development and Databricks Genie Agents.
• Lead two-week Agile sprints and client demonstrations.
• Conduct code/design reviews and knowledge transfer.

Skills

Data leadership

Azure Databricks

PySpark

SQL

Delta Lake

Unity Catalog

Azure DevOps

CI/CD

Agile leadership

Client communication

Tools

Azure Databricks

Azure DevOps

CI/CD pipelines

Oracle Fusion HCM/Finance

Job description

We are looking for an experienced Technical Lead – Azure Databricks to lead the delivery of a modern Azure Databricks data platform.

Responsibilities

• Lead the discovery phase covering source-system profiling, ingestion requirements and delivery planning.
• Own Azure Databricks platform architecture, configuration and technical delivery.
• Design environments, Unity Catalog governance, security, access controls, monitoring and cost controls.
• Lead development of metadata-driven ingestion frameworks and pipelines.
• Manage integration of on-premise SQL Server and Oracle Fusion HCM/Finance data.
• Establish CI/CD using Azure DevOps.
• Oversee semantic-layer development and Databricks Genie Agents.
• Lead two-week Agile sprints and client demonstrations.
• Conduct code/design reviews and knowledge transfer.

Qualifications

• 10+ years in Data Engineering/Data Platform delivery.
• Minimum 4+ years hands-on Azure Databricks.
• Led at least one complete Azure Databricks Lakehouse implementation from design through production.
• Strong PySpark, SQL, Delta Lake and Unity Catalog.
• On-premise database-to-Azure integration experience.
• Azure DevOps CI/CD and Databricks deployment experience.
• Experience leading remote/offshore Agile teams.
• Strong client-facing communication skills.
• Fluent English.
• UAE-based or willing to relocate to Abu Dhabi.

Preferred

• Delta Sharing, Lakeflow Connect/ADF, Genie Agents, Oracle Fusion HCM/Financials and UAE/GCC government experience.