Platform Architect / Platform Lead – Azure Databricks

Epergne Solutions; Abu Dhabi; On-site; AED 279,000 - 390,600; Full time

Job summary

A data solutions provider in Abu Dhabi is seeking a Platform Architect / Platform Lead specializing in Azure Databricks. The role entails owning the architecture, governance, and evolution of an enterprise-scale Data Intelligence Platform. Candidates should have over 7 years of experience, strong expertise in Azure Databricks, and a proven ability to design large-scale, secure data platforms. The position focuses on optimizing performance, ensuring scalability, and implementing cost governance throughout the platform.

Qualifications

• 7+ years of experience in data architecture and governance.
• Expertise in designing enterprise-grade data platforms.
• Strong knowledge of AI lifecycle management.

Responsibilities

• Define end-to-end Lakehouse architecture for Azure Databricks.
• Establish standards for declarative pipelines and workflows.
• Lead performance optimization for Spark and AI workloads.

Skills

Azure Databricks

Spark

PySpark

Lakehouse architecture

Unity Catalog

CI/CD governance

DevOps best practices

Job description

Job Location: Abu Dhabi, UAE

Experience: 7+ Years

Job Role

Job Role: Platform Architect / Platform Lead ? Azure Databricks

Role Summary

Own the end-to-end architecture, governance, security, and evolution of an enterprise-scale Azure Databricks Data Intelligence Platform, ensuring scalability, performance, cost optimization, and AI enablement.

Key Responsibilities

• Define end-to-end Lakehouse architecture covering ingestion, storage, processing, orchestration, analytics, and AI
• Design and govern Unity Catalog (catalogs, schemas, access control, lineage, auditing)
• Establish standards for declarative pipelines, workflows orchestration, and runtime governance
• Define governance framework for ML, Feature Store, AI capabilities, and analytics workloads
• Own Azure integrations including identity, networking, key management, and secure data access
• Implement compute, cluster, and cost governance (policies, sizing, concurrency, spend control)
• Review and approve architecture designs for security, scalability, and cost efficiency
• Oversee CI/CD governance and platform change management
• Lead performance optimization for Spark, streaming, and AI workloads
• Drive controlled adoption of new platform features and best practices
• Communicate architectural strategy to technical and business stakeholders

Required Skills & Experience

• Strong expertise in Azure Databricks, Spark, PySpark, and Lakehouse architecture
• Deep knowledge of Unity Catalog, Delta Lake, and platform governance frameworks
• Experience with orchestration, declarative pipelines, and workload optimization
• Hands-on exposure to ML lifecycle management and AI platform integration
• Strong understanding of Azure identity, networking, and security architecture
• Experience implementing cost governance and performance tuning strategies
• CI/CD governance and DevOps best practices experience
• Proven ability to design scalable, secure, enterprise-grade data platforms