Sr. Databricks Platform Engineer

Hiredge Solutions; Abu Dhabi; On-site; AED 40,000 - 60,000; Full time

Job summary

Hiredge Solutions is seeking a Senior Databricks platform engineer to lead the design of a governed multi-tenant data platform on Azure Databricks. The role involves governance implementation, architecting Delta Sharing, and contributing to the existing architecture.

The ideal candidate will have 7+ years in data platform engineering, strong experience in Unity Catalog, Delta Sharing, and production pipelines. This is an opportunity to work with a global consultancy on high-visibility projects.

Qualifications

• 7+ years in data platform engineering.
• 3+ years hands-on experience with Azure Databricks.
• Proven experience in a production environment.

Responsibilities

• Lead design and operationalisation of a multi-tenant data platform.
• Implement governance and architect Delta Sharing.
• Refactor Python pipelines into production-grade architecture.

Skills

Unity Catalog implementation

Delta Sharing experience

Strong Python

PySpark

Microsoft Entra ID integration

Data platform engineering

Tools

Azure Databricks

Azure Data Factory

Job description

The Role

We're engaging a hands‑on / Senior Databricks platform engineer/builder for our client - one of global leading strategy‑led tech consultancy company. The objective of role is to lead the design and operationalisation of a governed, multi‑tenant data platform on Azure Databricks - ingesting and onboarding data from multi external organisations across Databricks, Snowflake, API, and file‑based sources.

Design and implement governance from the ground up, and make Unity Catalog and Delta Sharing work at scale in production, contributing to the low level design of the architecture already selected.

What You’ll Be Doing

• Unity Catalog governance — design and implement the catalog/schema structure, privilege model, data ownership, and lineage for a platform serving multiple business units and external partners
• Delta Sharing — architect the provider/recipient model for external data onboarding; manage refresh strategy, access scoping, and cross‑platform sharing (Databricks-to-Databricks, Databricks-to-Snowflake)
• Identity & access integration — Microsoft Entra ID integration with Unity Catalog; service principal design; RBAC across environment tiers
• Pipeline modernisation — refactor existing Python pipelines into a production‑grade Databricks medallion architecture (bronze/silver/gold)
• Platform engineering — workspace structure, environment promotion, CI/CD, cost optimisation, and capacity planning
• Architecture contribution — this is a live engagement with an existing recommendation in place; we want someone who will challenge and improve it, not just implement it

What you should have:

Must‑Have

• Hands‑on Unity Catalog implementation in a production environment — not workshop‑level exposure. You should be able to talk through catalog structure, privilege model, and lineage from real deployments.
• Real Delta Sharing experience — provider/recipient architecture, external data onboarding, operational challenges you’ve actually hit and solved
• Strong Python and PySpark, with production pipeline migration experience (not just greenfield builds)
• Microsoft Entra ID integration experience — service principals, RBAC, scoped access
• Track record of making platform architecture decisions yourself, not just executing someone else’s design
• 7+ years data platform engineering, 3+ years hands‑on Azure Databricks

Nice to Have

• Databricks Genie evaluation/implementation experience
• Azure Data Factory orchestration
• CI/CD and DataOps (Azure DevOps, Databricks Asset Bundles)
• FinOps / platform cost optimisation
• Regulated industry or enterprise governance experience

If you’re an experienced builder in applying Unity Catalog & Delta Sharing to external Data integration, and looking to work alongside a top‑tier global consultancy on a high visibility platform build / product‑grade delivery, we’d like to hear from you.