Azure Databricks Solution Architect – Data Platform Lead

EXL; New Jersey; On-site; USD 140,000 - 190,000; Full time

Job summary

EXL is seeking an experienced data architecture lead to design and govern enterprise data platforms on Azure, with a focus on Databricks Lakehouse, Delta Lake, and scalable cloud-native ecosystems.

The role involves defining ingestion, transformation, and serving layers for reporting, analytics, ML, and critical decisioning use cases, as well as building robust ETL/ELT pipelines and establishing governance and release processes across Databricks solutions.

Qualifications

• Experience designing enterprise data platforms on Azure with Databricks Lakehouse architectures.
• Strong knowledge of PySpark, SQL, Databricks workflows and Delta Live Tables.
• Experience with data governance, security controls, Unity Catalog and Purview integration.
• Familiarity with CI/CD, environment strategy and release management for data platforms.

Responsibilities

• Lead end-to-end architecture and solution design for enterprise data platforms on Azure, focusing on Databricks Lakehouse.
• Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers for analytics and ML use cases.
• Design and implement robust ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables.
• Own architecture standards including data modeling, medallion design, reusable patterns, and release governance across Databricks solutions.
• Drive platform governance and security with Unity Catalog, RBAC/ABAC, lineage, and enterprise governance integrations.
• Optimize Spark performance through tuning, partitioning, caching and cost management for large-scale processing.
• Collaborate with stakeholders to translate requirements into scalable designs and provide technical leadership to engineering teams.
• Evaluate features like Photon, serverless compute, Lakehouse Federation and streaming patterns to improve scalability and value.

Job description

EXL is seeking an experienced data architecture lead to design and govern enterprise data platforms on Azure, with a focus on Databricks Lakehouse, Delta Lake, and scalable cloud-native ecosystems.

The role involves defining ingestion, transformation, and serving layers for reporting, analytics, ML, and critical decisioning use cases, as well as building robust ETL/ELT pipelines and establishing governance and release processes across Databricks solutions.