Data Engineer

EY; Ahmedabad District; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

EY is seeking a senior data engineering lead to design Lakehouse architectures on Azure Databricks, leveraging Delta Lake and Unity Catalog to deliver enterprise data products.

You will define medallion patterns and architect ingestion frameworks across batch and streaming data, ensuring governance and security with RBAC and Key Vault. Mentorship and collaboration with Cyber, Cloud, and IAM teams are essential.

Qualifications

• Deep expertise in Azure Databricks, Delta Lake, Unity Catalog, Spark (PySpark/SQL).
• Strong on Azure services: ADLS Gen2, ADF, Synapse, Key Vault, Entra ID, Event Hubs, Functions.
• Proven experience designing medallion/lakehouse architectures at enterprise scale.

Responsibilities

• Design Lakehouse architectures on Azure Databricks using Delta Lake and Unity Catalog.
• Define medallion (Bronze/Silver/Gold) patterns, data models, and enterprise data products.
• Architect ingestion frameworks (batch, streaming, CDC) using Auto Loader, ADF, Event Hubs, Kafka.

Skills

Azure Databricks

Delta Lake

Unity Catalog

Spark PySpark/SQL

Data modeling

CI/CD / DevOps

Governance & security

Azure services

Git

IaC (Terraform/Bicep)

Tools

Terraform/Bicep

Git

Azure DevOps / GitHub Actions

Job description

Role & responsibilities

• Design Lakehouse architectures on Azure Databricks using Delta Lake and Unity Catalog.
• Define medallion (Bronze/Silver/Gold) patterns, data models, and enterprise data products.
• Architect ingestion frameworks (batch, streaming, CDC) using Auto Loader, ADF, Event Hubs, Kafka.
• Establish governance, security, and RBAC via Unity Catalog, Key Vault, and service principals.
• Optimize performance and cost (cluster sizing, Photon, Liquid Clustering, Z-Order, caching).
• Lead CI/CD with Databricks Asset Bundles, Git, and Azure DevOps / GitHub Actions.
• Guide teams on DLT, MLflow, Delta Sharing, and Lakehouse Monitoring.
• Partner with Cyber, Cloud, and IAM teams on approvals, allow-listing, and compliance.
• Mentor data engineers and review designs, code, and production readiness.

Must-Have Skills

• Deep expertise in Azure Databricks, Delta Lake, Unity Catalog, Spark (PySpark/SQL).
• Strong on Azure services: ADLS Gen2, ADF, Synapse, Key Vault, Entra ID, Event Hubs, Functions.
• Proven experience designing medallion / lakehouse architectures at enterprise scale.
• Solid grounding in data modeling (dimensional, Data Vault, or 3NF) and SQL performance tuning.
• CI/CD, IaC (Terraform/Bicep), Git-based workflows, and DevOps practices.
• Data governance, lineage, security, and privacy controls in regulated environments