Databricks Solution Architect

EXL; New Jersey; On-site; USD 140,000 - 190,000; Full time

Job summary

EXL is seeking an experienced data architecture lead to design and govern enterprise data platforms on Azure, with a focus on Databricks Lakehouse, Delta Lake, and scalable cloud-native ecosystems.

The role involves defining ingestion, transformation, and serving layers for reporting, analytics, ML, and critical decisioning use cases, as well as building robust ETL/ELT pipelines and establishing governance and release processes across Databricks solutions.

Qualifications

• Experience designing enterprise data platforms on Azure with Databricks Lakehouse architectures.
• Strong knowledge of PySpark, SQL, Databricks workflows and Delta Live Tables.
• Experience with data governance, security controls, Unity Catalog and Purview integration.
• Familiarity with CI/CD, environment strategy and release management for data platforms.

Responsibilities

• Lead end-to-end architecture and solution design for enterprise data platforms on Azure, focusing on Databricks Lakehouse.
• Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers for analytics and ML use cases.
• Design and implement robust ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables.
• Own architecture standards including data modeling, medallion design, reusable patterns, and release governance across Databricks solutions.
• Drive platform governance and security with Unity Catalog, RBAC/ABAC, lineage, and enterprise governance integrations.
• Optimize Spark performance through tuning, partitioning, caching and cost management for large-scale processing.
• Collaborate with stakeholders to translate requirements into scalable designs and provide technical leadership to engineering teams.
• Evaluate features like Photon, serverless compute, Lakehouse Federation and streaming patterns to improve scalability and value.

Job description

• Lead the end-to-end architecture and solution design for enterprise data platforms on Azure, with strong focus on Databricks Lakehouse, Delta Lake, and scalable cloud-native data ecosystems.
• Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers to support reporting, advanced analytics, ML, and business-critical decisioning use cases.
• Design and implement robust ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables for batch and near real-time processing.
• Own architecture standards for data modeling, medallion design, reusable engineering patterns, CI/CD, code quality, environment strategy, and release management across Databricks solutions.
• Drive platform governance and security using Unity Catalog, RBAC/ABAC controls, lineage, auditability, and integration with enterprise governance services such as Purview.
• Optimize solution performance by tuning Spark workloads, cluster policies, partitioning strategy, file sizing, caching, and compute cost management for large-scale data processing.
• Collaborate with business stakeholders, product owners, analysts, architects, and downstream consumers to translate functional and non-functional requirements into scalable technical designs.
• Provide technical leadership to engineering teams by reviewing designs, guiding implementation, resolving architectural bottlenecks, and establishing best practices for Databricks-based delivery.
• Evaluate and recommend Databricks capabilities such as Photon, serverless compute, Lakehouse Federation, and streaming patterns to improve scalability, maintainability, and time to value.
• Ensure strong delivery governance through estimation, technical planning, dependency management, risk mitigation, and Agile execution including sprint planning, backlog refinement, and design reviews.