IN_Senior Associate_Azure Data Bricks_Digital Integration_Advisory_Kolkata

Price Waterhouse Cooper LLP; Kolkata District; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Price Waterhouse Cooper LLP in Kolkata, India seeks an Azure Databricks Data Engineer to design, develop, and optimize scalable data platforms on Microsoft Azure. You will build data pipelines using Databricks, PySpark, Spark SQL, and Delta Lake, collaborating with stakeholders to deliver enterprise-grade solutions while ensuring data quality and governance.

The role requires strong Azure data services experience, SQL proficiency, and hands-on development in an Agile environment.

Qualifications

• Strong experience with Azure Databricks.
• Hands-on PySpark, Spark SQL, and Delta Lake.
• Experience with Lakehouse architecture and large-scale data processing.
• Understanding of Spark performance tuning and optimization techniques.
• Experience with Azure Data Factory (ADF), ADLS Gen2, Azure SQL Database, Azure Key Vault.
• Strong experience in ETL/ELT development.
• Advanced SQL and T-SQL proficiency.
• Experience with PostgreSQL databases.
• Knowledge of data warehousing concepts and dimensional modeling.
• Experience in data architecture and data modeling.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Develop ETL/ELT workflows using PySpark, Spark SQL, and Azure Data Factory.
• Build and manage Lakehouse architectures using Delta Lake and ADLS Gen2.
• Implement data ingestion frameworks for structured and unstructured data sources.
• Optimize Spark jobs, cluster performance, and query execution.
• Translate requirements into data solutions with business teams.
• Design and maintain data models, schemas, and data integration processes.
• Ensure data quality, governance, security, and compliance standards.
• Troubleshoot production issues and optimize data processing performance.
• Create and maintain technical documentation and procedures.
• Participate in Agile methodologies including Scrum, Kanban, or SAFe.

Skills

Azure Databricks

PySpark

Delta Lake

Azure Data Factory

ADLS Gen2

Azure SQL Database

Azure Key Vault

SQL

PostgreSQL

Python

Spark SQL

ETL/ELT development

Data Warehousing

Lakehouse

Data Governance

Education

Bachelor's Degree in CS/IT/Data Engineering

Microsoft Certified: Azure Solutions Architect Expert

Tools

Unity Catalog

Microsoft Purview

Snowflake

Power BI

Structured Streaming

Job description

3 - 5 Years

Regular

Job Description

We are seeking an Azure Databricks Data Engineer to design, develop, and optimize scalable data platforms and analytics solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure Databricks, PySpark, Delta Lake, and data engineering best practices to build efficient, reliable, and high-performance data pipelines.

The role involves collaborating with business stakeholders, data architects, and analytics teams to deliver enterprise-grade data solutions while ensuring data quality, governance, scalability, and performance.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.


• Develop ETL/ELT workflows using PySpark, Spark SQL, and Azure Data Factory.


• Build and manage Lakehouse architectures using Delta Lake and Azure Data Lake Storage (ADLS Gen2).


• Implement data ingestion frameworks for structured and unstructured data sources.


• Optimize Spark jobs, cluster performance, and query execution.


• Work with business and technical teams to translate requirements into data solutions.


• Design and maintain data models, schemas, and data integration processes.


• Ensure data quality, governance, security, and compliance standards.


• Troubleshoot production issues and optimize data processing performance.


• Create and maintain technical documentation and operational procedures.


• Participate in Agile methodologies including Scrum, Kanban, or SAFe.

Mandatory Skill Sets

• Strong experience with Azure Databricks .


• Hands-on expertise in PySpark , Spark SQL , and Delta Lake .


• Experience with Lakehouse architecture and large-scale data processing.


• Understanding of Spark performance tuning and optimization techniques.


• Experience with:
  
  
  
  • Azure Data Factory (ADF)
  
  
  • Azure Data Lake Storage Gen2 (ADLS)
  
  
  • Azure SQL Database
  
  
  • Azure Key Vault
  
  
  
  


• Strong experience in ETL/ELT development.


• Advanced SQL and T‑SQL proficiency.


• Experience with PostgreSQL databases.


• Knowledge of data warehousing concepts and dimensional modeling.


• Experience in data architecture and data modeling.

Programming

• Strong proficiency in Python and PySpark .

Development Methodology

• Experience working in Agile environments (Scrum, Kanban, or SAFe).

Good to Have Skill Sets

• Unity Catalog


• Microsoft Purview


• Snowflake


• Power BI


• Structured Streaming

Preferred Skill Sets

• Strong hands‑on experience with Azure Databricks, PySpark, Python, and SQL.


• Expertise in ETL, ELT, Data Warehousing, and Lakehouse architectures.


• Experience with data governance, metadata management, and lineage.


• Knowledge of performance optimization and cost management within Azure Databricks.

Years of Experience Required

4-7 years of exp.

Certifications / Credentials (Any One Mandatory)

• Microsoft Certified: Azure Solutions Architect Expert

Educational Qualification

Bachelor's Degree in Computer Science, Information Technology, Data Engineering, or a related field .