Consultant | Databricks + Pyspark | Mumbai | Engineering | Data Modernization & Migration

Deloitte & Touche GmbH Wirtschaftsprüfungsgesellschaft; Mumbai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Deloitte & Touche GmbH Wirtschaftsprüfungsgesellschaft in Mumbai seeks a Databricks Consultant to design and implement data engineering solutions on the Azure Databricks platform. The role focuses on building scalable ETL/ELT pipelines, transforming data with PySpark, and organizing data within Delta Lake for downstream analytics.

Candidates should have 3–5 years in data engineering with Azure Databricks, strong SQL skills, and experience with ADLS Gen2 and ADF.

Qualifications

• Bachelor’s degree in Computer Science or related field.
• Experience with Databricks on Azure Databricks platform.
• Strong PySpark and SparkSQL experience for data transformation.
• Proficiency with SQL and relational databases (SQL Server, Oracle, PostgreSQL).
• Experience with Azure Data Lake Storage (ADLS Gen2) and Azure Data Factory (ADF).

Responsibilities

• Develop and maintain ETL/ELT pipelines using AzureDatabricks to move and transform data.
• Implement data processing workflows using PySpark and SparkSQL within Databricks notebooks.
• Manage data within Delta Lake with schema enforcement.
• Use Azure Data Factory to orchestrate Databricks notebooks and workflows.
• Integrate data sources (SQL Server, Oracle, etc.) into the Databricks ecosystem.
• Support data cataloging and documentation for data discoverability.
• Troubleshoot Spark jobs and data pipelines to ensure data quality.

Skills

Problem solving

Analytical thinking

Agile teamwork

Communication skills

Education

Bachelor’s degree in Computer Science

Tools

Databricks

PySpark

SparkSQL

Delta Lake

Azure Data Factory

ADLS Gen2

SQL Server

Oracle

PostgreSQL

Job description

Select how often (in days) to receive an alert:

Job Title: Consultant | Databricks + Pyspark | Mumbai | Engineering | Data Modernization & Migration

Job Title: Databricks Consultant

Role Overview: As a Databricks Consultant, you will be responsible for developing and implementing data engineering solutions on the Azure Databricks platform. Your role involves designing and maintaining scalable data pipelines, performing data transformations using Spark, and ensuring data is properly structured within a Delta Lake environment for downstream analytics.

Experience:

• 3–5 years of experience in data engineering, with a strong emphasis on AzureDatabricks.
• Proficiency in PySpark or SparkSQL for data transformation.
• Experience with Azure Data Lake Storage (ADLS Gen2) and Azure Data Factory (ADF).
• Strong foundation in SQL and relational databases (SQL Server, Oracle, or PostgreSQL).

Key Responsibilities: Technical Skills:

• Develop and maintain ETL/ELT pipelines using AzureDatabricks to move and transform data across diverse sources.
• Implement data processing workflows using PySpark and SparkSQL within Databricks notebooks.
• Manage and organize data within Delta Lake, utilizing features like Delta tables and schema enforcement.
• Use Azure Data Factory (ADF) to orchestrate Databricks notebooks and workflows.
• Integrate various data sources (SQL Server, Oracle, etc.) into the Azure Databricks ecosystem.
• Support data cataloging and documentation efforts to ensure data discoverability.
• Troubleshoot and resolve issues related to Spark jobs, data pipelines, and data quality.

Soft Skills:

• Strong problem-solving and analytical abilities.
• Ability to work effectively within agile, cross-functional teams.
• Strong communication skills for technical documentation and team collaboration.

Preferred Qualifications:

• Bachelor’s degree in Computer Science or a related field.
• Databricks Certified Data Engineer Associate is highly desirable.