Databricks - Azure Data engineer

PwC India; India; Hybrid; INR 1,400,000 - 2,200,000; Full time

Job summary

Price Waterhouse Cooper LLP in Pan India seeks a Databricks Developer with 4–7 years of experience to design, develop, and optimize data engineering solutions on the Databricks Lakehouse Platform on Azure.

You will build scalable data pipelines, implement Delta Lake architectures, and ensure governed, cost-efficient data platforms. Strong PySpark, Spark SQL, and Azure Data Factory skills are required for this role.

Qualifications

• Databricks and Spark-based data engineering expertise.
• Experience with Delta Live Tables and Unity Catalog.
• Proficient in PySpark, SQL, and Azure Data Factory.

Responsibilities

• Design and develop data pipelines using Spark, PySpark, and Spark SQL in Databricks.
• Build Delta Live Tables pipelines for reliable ETL processing.
• Implement Structured Streaming and Auto Loader for real-time ingestion.
• Develop workflows with Databricks Workflows/Jobs and Azure Data Factory.
• Design Delta Lake / Lakehouse architecture Bronze/Silver/Gold patterns.
• Configure Unity Catalog for data governance and access control.
• Create analytical layers using Databricks SQL Warehouses for BI/reporting.
• Manage data storage/movement across ADLS Gen2 and Azure SQL.
• Write production-grade Python scripts for data transformations and automation.
• Optimize Spark jobs for partitioning, caching, and shuffle.

Skills

Databricks

Delta Lake

Spark PySpark

Streaming

Python

Azure Data Factory

Azure SQL

ADLS Gen2

Performance Tuning

Education

Bachelor's degree in CS/IT/Engineering

Tools

Databricks SQL Warehouses

Unity Catalog

Delta Live Tables

GitHub Actions

Azure DevOps

Job description

Experience Required: 4-7 years (Senior Associate)

Location: Pan India

Job Title: Databricks Developer

Role Overview: We are seeking an experienced Databricks Developer to design, develop, and optimize data engineering solutions on the Databricks Lakehouse Platform. The role involves building scalable data pipelines, implementing Delta Lake architectures, and ensuring high-performance, governed, and cost-efficient data platforms on Azure.

Key Responsibilities:

• Design and develop data pipelines using Apache Spark, PySpark, and Spark SQL within Databricks Workspace.
• Build and manage Delta Live Tables (DLT) pipelines for declarative, reliable ETL processing.
• Implement Structured Streaming and Auto Loader for incremental and real-time data ingestion.
• Develop and orchestrate workflows using Databricks Workflows/Jobs and Azure Data Factory (ADF).
• Design and maintain Delta Lake & Lakehouse Architecture Bronze/Silver/Gold layered patterns.
• Configure and manage Unity Catalog for centralized data governance, lineage, and access control.
• Build analytical layers using Databricks SQL Warehouses for BI and reporting consumption.
• Manage data storage and movement across ADLS Gen2 and Azure SQL environments.
• Write production-grade Python scripts for data transformations, utilities, and automation.
• Optimize Spark jobs for performance — partitioning, caching, broadcast joins, shuffle optimization.

Required Skills:

• Databricks (Must-Have): 3+ years — Workspace, Unity Catalog, Delta Live Tables (DLT), Workflows/Jobs, SQL Warehouses
• Delta Lake & Lakehouse: Lakehouse architecture design, Bronze/Silver/Gold patterns, ACID transactions, schema evolution
• Spark & PySpark (Must-Have): DataFrame API, Spark SQL, transformations, actions, performance tuning
• Streaming (Must-Have): Structured Streaming, Auto Loader for incremental ingestion
• Python (Must-Have): Production-grade scripting for data engineering and automation
• Azure Stack (Must-Have) ADF (orchestration), ADLS Gen2 (storage), Azure SQL (relational)
• Performance Tuning: Partitioning, Z-ordering, caching, shuffle optimization, job cluster sizing

Preferred Qualifications:

• Experience with Databricks Asset Bundles or CI/CD for Databricks (GitHub Actions, Azure DevOps).
• Familiarity with data modeling concepts (Star Schema, Data Vault 2.0).
• Exposure to Power BI / Tableau connecting via Databricks SQL.
• Knowledge of Databricks cost optimization (cluster policies, spot instances, photon engine).
• Databricks Certified Data Engineer Associate/Professional is a plus.
• Exposure to Agile/Scrum delivery methodologies.

Qualifications:

• Bachelor's degree in CS/IT/Engineering.
• 3–7 years experience with minimum 3+ years in Databricks and Spark-based data engineering.
• Consulting/services firm experience preferred.