Data Engineering Assistant Vice President

EXL; Dadri, Bengaluru, Delhi; Hybrid; INR 2,500,000 - 5,000,000; Full time

Job summary

EXL is seeking a Senior Databricks Architect to design and lead Lakehouse architecture for large-scale healthcare data platforms. You will define scalable batch and real-time data processing, ingestion, transformation, storage, and consumption patterns using Databricks and Delta Lake.

You will guide governance through Unity Catalog, Delta Live Tables, and workflows, while ensuring secure handling of PHI/PII and collaborating across Azure, AWS, or GCP.

Qualifications

• Hands-on Databricks experience across large-scale healthcare data platforms.
• Expertise in Lakehouse architecture and Delta Lake.
• Strong PySpark and SQL skills.
• Experience with Unity Catalog and data governance.
• Cloud platform experience (Azure/AWS/GCP).
• CI/CD, Git, and DevOps practices.

Responsibilities

• Design and lead Databricks Lakehouse architecture for large-scale healthcare data platforms.
• Define scalable data architectures for batch and real-time healthcare data processing.
• Design data ingestion, transformation, storage, and consumption patterns using Databricks.
• Develop architecture using Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables/Lakeflow.
• Integrate data from multiple healthcare sources, including clinical, claims, eligibility, provider, and patient systems.
• Design robust ETL/ELT pipelines using PySpark, SQL, and Databricks.
• Establish data quality, validation, reconciliation, lineage, and governance frameworks.
• Design secure healthcare data solutions while considering HIPAA and PHI/PII protection requirements.
• Work with cloud platforms such as Azure, AWS, or GCP to build scalable data solutions.
• Collaborate with data engineers, data scientists, analysts, product owners, and business stakeholders.
• Define technical standards, architectural patterns, and best practices for Databricks development.
• Lead performance optimization and cost-management initiatives across the Databricks platform.
• Provide technical leadership and mentoring to data engineering teams.
• Participate in Agile ceremonies and contribute to technical planning and roadmap discussions.

Skills

Databricks

Lakehouse architecture

PySpark

SQL

Unity Catalog

Data governance

Cloud platforms

CI/CD

Git

DevOps

Tools

Delta Lake

Delta Live Tables

Databricks Workflows

Lakeflow

CI/CD tooling

Job description

Responsibilities

• Design and lead Databricks Lakehouse architecture for large-scale healthcare data platforms.
• Define scalable data architectures for batch and real-time healthcare data processing.
• Design data ingestion, transformation, storage, and consumption patterns using Databricks.
• Develop architecture using Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables/Lakeflow.
• Integrate data from multiple healthcare sources, including clinical, claims, eligibility, provider, and patient systems.
• Design robust ETL/ELT pipelines using PySpark, SQL, and Databricks.
• Establish data quality, validation, reconciliation, lineage, and governance frameworks.
• Design secure healthcare data solutions while considering HIPAA and PHI/PII protection requirements.
• Work with cloud platforms such as Azure, AWS, or GCP to build scalable data solutions.
• Collaborate with data engineers, data scientists, analysts, product owners, and business stakeholders.
• Define technical standards, architectural patterns, and best practices for Databricks development.
• Lead performance optimization and cost-management initiatives across the Databricks platform.
• Provide technical leadership and mentoring to data engineering teams.
• Participate in Agile ceremonies and contribute to technical planning and roadmap discussions.

Qualifications

• Strong hands-on experience with Databricks.
• Expertise in Lakehouse architecture and Delta Lake.
• Strong PySpark and SQL skills.
• Experience with Unity Catalog and data governance.
• Experience designing enterprise-scale data platforms.
• Strong knowledge of ETL/ELT and data modeling.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Experience with data integration and orchestration tools.
• Knowledge of CI/CD, Git, and DevOps practices.
• Strong understanding of data security and access control.
• Experience with Agile/Scrum methodologies.