Senior Associate _Azure Databricks Pyspark Developer -_Data & Analytics _Advisory _Mumbai & Pune

PwC India; Mumbai; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

PwC India is seeking an IN_Senior Associate for Azure Databricks PySpark development in Data & Analytics Advisory, supporting Pune & Mumbai operations.

The role focuses on building Data Lake/Lakehouse architectures, scalable pipelines, and governance using ADLS, Unity Catalog, and Delta Live Tables. Collaboration with data scientists and mentors junior team members is expected.

Qualifications

• Data Lake and Lakehouse architecture design, implementation and management.
• Develop and maintain scalable data pipelines and workflows.
• Utilize Azure Data Lake Services (ADLS) for data storage and management.
• Knowledge on Medalion Architecture, Delta Format.
• Data Processing and Transformation using PySpark.
• Implement Delta Live Tables for real-time processing (good to have).
• Ensure data quality and governance across data lifecycle.
• Extract, transform, and load data from multiple sources including SAP and Dynamics 365.

Responsibilities

• Data Lake and Lakehouse implementation and ongoing optimization.
• Develop data pipelines and orchestration with ADLS, ADF, and Synapse.
• Ensure data security, governance, and metadata management.
• Collaborate with data scientists and analysts to meet data requirements.
• Provide technical guidance to junior members and drive continuous improvement.

Skills

PySpark

Python

Databricks

Azure Data Lake Services

Unity Catalog

Delta Live Tables

Azure Data Factory

Synapse Analytics

SAP

Dynamics 365

Azure Fabric

Data Lake architecture

Education

B.E. / B.Tech / MCA/ M.E/ M.TECH/ MBA/ PGDM

Tools

SAP

Dynamics 365

Databricks

Azure Data Factory

Synapse Analytics

Job description

Job Description & Summary

A career within Technology Consulting services, will provide you with the opportunity to bring our clients a competitive advantage through defining their technology objectives, assessing solution options, and devising architectural solutions that help them achieve both strategic goals and meet operational requirements. We help build software and design data platforms, manage large volumes of client data, develop compliance procedures for data management, and continually researching new technologies to drive innovation and sustainable change.

Job Position Title

IN_Senior Associate _Azure Databricks Pyspark Developer -_Data & Analytics _Advisory _Pune & Mumbai

Responsibilities

Key Responsibilities

• Data Lake and Lakehouse Implementation:
• Design, implement, and manage Data Lake and Lakehouse architectures. (Must have)
• Develop and maintain scalable data pipelines and workflows. (Must have)
• Utilize Azure Data Lake Services (ADLS) for data storage and management. (Must have)
• Knowledge on Medalion Architecture, Delta Format. (Must have)
• Data Processing and Transformation:
• Use PySpark for data processing and transformations. (Must have)
• Implement Delta Live Tables for real-time data processing and analytics. (Good to have)
• Ensure data quality and consistency across all stages of the data lifecycle. (Must have)
• Data Management and Governance:
• Employ Unity Catalog for data governance and metadata management. (Good to have)
• Ensure robust data security and compliance with industry standards. (Must have)
• Data Integration:
• Extract, transform, and load (ETL) data from multiple sources (Must have) including SAP (Good to have), Dynamics 365 (Good to have), and other systems.
• Utilize Azure Data Factory (ADF) and Synapse Analytics for data integration and orchestration. (Must have)
• Performance Optimization of the Jobs. (Must have)
• Data Storage and Access:
• Implement and manage Azure Data Lake Storage (ADLS) for large-scale data storage. (Must have)
• Optimize data storage and retrieval processes for performance and cost-efficiency. (Must have)
• Collaboration and Communication:
• Work closely with data scientists, analysts, and other stakeholders to understand data requirements. (Must have)
• Provide technical guidance and mentorship to junior team members. (Good to have)
• Continuous Improvement:
• Stay updated with the latest industry trends and technologies in data engineering and cloud computing. (Good to have)
• Continuously improve data processes and infrastructure for efficiency and scalability. (Must have)

Mandatory skill sets

• Technical Skills:
• Proficient in PySpark and Python for data processing and analysis.
• Strong experience with Azure Data Lake Services (ADLS) and Data Lake architecture.
• Hands-on experience with Databricks for data engineering and analytics.
• Knowledge of Unity Catalog for data governance.
• Expertise in Delta Live Tables for real-time data processing.
• Familiarity with Azure Fabric for data integration and orchestration.
• Proficient in Azure Data Factory (ADF) and Synapse Analytics for ETL and data warehousing.
• Experience in pulling data from multiple sources like SAP, Dynamics 365, and others.
• Soft Skills:
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.
• Ability to work independently and as part of a team.
• Attention to detail and commitment to data accuracy and quality.

Preferred skill sets

Handson ETL / Datalake is good to have..

Certifications required

• Certification in Azure Data Engineering or relevant Azure certifications.
• DP203 (Must have)
• Certification in Databricks.
• Databricks certified Data Engineer Associate (Must have)
• Databricks certified Data Engineer Professional (Good Have)

Years of experience required

2-4 Years

Educational Qualification

B.E. / B.Tech / MCA/ M.E/ M.TECH/ MBA/ PGDM. All qualifications should be in regular full-time mode with no extension of course duration due to backlogs.