Senior Consultant | Databricks + Pyspark | Mumbai | Engineering | Data Modernization & Migration

Deloitte & Touche GmbH Wirtschaftsprüfungsgesellschaft; Mumbai; On-site; INR 2,600,000 - 3,800,000; Full time

Job summary

Deloitte & Touche GmbH Wirtschaftsprüfungsgesellschaft in Mumbai seeks a Senior Consultant to design and optimize data processing frameworks on Azure Databricks. You will drive Medallion architectures with Delta LiveTables, oversee Spark tuning, and enforce data governance with Unity Catalog.

You will mentor junior engineers, lead CI/CD for Databricks, and architect data integration across Databricks, Azure Synapse, and RDBMS while optimizing cloud costs.

Qualifications

• 5–7 years of data engineering experience with Azure Databricks and Apache Spark.
• Experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold).
• Proven Spark performance tuning experience (data skew, shuffle tuning).
• Strong Unity Catalog experience for data governance and security.

Responsibilities

• Design and implement scalable Medallion architectures using Delta LiveTables and Databricks.
• Lead the optimization of complex Spark workloads to improve execution speed and reduce cloud compute costs.
• Implement enterprise-grade data governance and access control using Unity Catalog.
• Architect automated CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs) or Terraform.
• Design sophisticated data integration patterns between Databricks, Azure Synapse, and various RDBMS.
• Act as a technical mentor to junior engineers, conducting code reviews and enforcing best practices.

Skills

Leadership

Problem-solving

Communication

Mentoring

Education

Bachelor's or Master’s degree in a technical field

Tools

Azure Databricks

Apache Spark

Delta Lake

Terraform

Unity Catalog

Job description

Select how often (in days) to receive an alert:

Job Title: Senior Consultant | Databricks + Pyspark | Mumbai | Engineering | Data Modernization & Migration

Job Title: Databricks - Sr. Consultant

Role Overview: As a Senior Consultant, you will lead the design and optimization of high-performance data processing frameworks on Azure Databricks. You will be responsible for advanced Spark tuning, implementing robust data governance through Unity Catalog, and driving the transition toward a modern Medallion architecture.

Experience:

• 5–7 years of experience in data engineering, with deep expertise in AzureDatabricks and Apache Spark.
• Advanced experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold).
• Proven track record in Spark Performance Tuning (handling data skew, shuffle tuning, etc.).
• Strong experience with Unity Catalog for data governance and security.

Key Responsibilities: Technical Skills:

• Design and implement scalable Medallion Architectures using Delta LiveTables (DLT) and Databricks.
• Lead the optimization of complex Spark workloads to improve execution speed and reduce cloud compute costs.
• Implement enterprise-grade data governance and access control using Unity Catalog.
• Architect automated CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs) or Terraform.
• Design sophisticated data integration patterns between Databricks, Azure Synapse, and various RDBMS.
• Act as a technical mentor to junior engineers, conducting code reviews and enforcing best practices.

Soft Skills:

• Strong leadership and ability to drive technical delivery.
• Advanced problem-solving skills for complex architectural challenges.
• Excellent communication skills for interacting with stakeholders and technical leads.

Preferred Qualifications:

• Bachelor’s or Master’s degree in a technical field.
• Databricks Certified Data Engineer Professional is highly desirable.
• Experience with DataOps and Infrastructure as Code (Terraform).