IN_Senior Associate_Databricks Data Engineer_GCC_Advisory_Bangalore

PricewaterhouseCoopers Services LLP

Bengaluru

On-site

INR 1,800,000 - 2,400,000

Full time

4 days ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

PwC is seeking an experienced Data Engineer – Databricks to design, build, and operate scalable data pipelines on the Databricks Lakehouse Platform. You will work with Spark, Delta Lake, and cloud services, collaborating with analytics and AI/ML teams to deliver data-driven insights.

Role requires hands-on development, data governance awareness, and mentoring junior engineers. 4+ years in data engineering and strong SQL are expected; travel and agile team participation are common in this

Qualifications

4+ years of experience as a Data Engineer with Databricks expertise.
Hands-on experience with Apache Spark, PySpark, and Spark SQL.
Strong knowledge of Delta Lake and Lakehouse architecture.
Advanced SQL skills and experience with ETL/ELT patterns and data warehousing.

Responsibilities

Design, build, and maintain end-to-end data pipelines using Databricks (PySpark / Spark SQL).
Ingest, transform, and curate large-scale structured and semi-structured datasets.
Collaborate with analytics, AI/ML, and business teams to support data products.
Provide production support and optimize Spark jobs for performance and cost.

Skills

Databricks

Spark

PySpark

Spark SQL

Delta Lake

SQL

ETL/ELT

Cloud platforms

Agile

Data governance

Education

Bachelor's or Master's in Computer Science or Engineering

Tools

Snowflake

Unity Catalog

Auto Loader

Python

CI/CD for data pipelines

Job description

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Senior Associate Job Description & Summary

At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals. In data analysis at PwC, you will focus on utilising advanced analytical techniques to extract insights from large datasets and drive data-driven decision-making. You will leverage skills in data manipulation, visualisation, and statistical modelling to support clients in solving complex business problems.

We reward your contributions, support your wellbeing, and offer inclusive benefits, flexibility programmes and mentorship that will help you thrive in work and life.

At PwC, we believe in providing equal employment opportunities, without any discrimination on the grounds of gender, ethnic background, age, disability, marital status, sexual orientation, pregnancy, gender identity or expression, religion or other beliefs, perceived differences and status protected by law. We strive to create an environment where each one of our people can bring their true selves and contribute to their personal growth and the firm’s growth. To enable this, we have zero tolerance for any discrimination and harassment based on the above considerations.

Job Description & Summary

We are seeking an experienced Data Engineer – Databricks to design, build, and operate scalable, high-performance data pipelines on the Databricks Lakehouse Platform. The role involves hands‑on development using Apache Spark, Databricks notebooks, Delta Lake, and cloud‑native services, along with close collaboration with analytics, AI/ML, and business teams.

Responsibilities

Design, build, and maintain end-to-end data pipelines using Databricks (PySpark / Spark SQL)
Implement batch and incremental data processing using Delta Lake and multi-hop architecture
Develop and optimize Databricks notebooks, jobs, and workflows
Ingest, transform, and curate large-scale structured and semi-structured datasets
Support analytics, reporting, and downstream data consumption use cases
Ensure data quality, reliability, lineage, and governance
Collaborate with data scientists, analysts, and architects on AI/ML workloads
Optimize Spark jobs for performance and cost efficiency
Adhere to enterprise security, access control, and compliance standards
Provide production support and troubleshoot data pipeline issues
Document technical designs, data flows, and operational runbooks
Mentor junior engineers and contribute to best practices

Mandatory skill sets

4+ years of experience as a Data Engineer with strong Databricks expertise
Hands‑on experience with Apache Spark, PySpark, and Spark SQL
Strong knowledge of Delta Lake and Lakehouse architecture
Advanced SQL skills
Experience with ETL/ELT patterns and data warehousing concepts
Exposure to at least one cloud platform (Azure / AWS / GCP)
Understanding of distributed computing concepts
Experience working in Agile teams

Preferred skill sets

Experience with Unity Catalog and data governance
Exposure to Auto Loader or streaming frameworks
CI/CD for data pipelines
Python for data engineering and automation
Databricks certification (Associate / Professional)

Years of experience required: 4-7 years

Education qualification: Bachelor’s or Master’s degree in Computer Science, Engineering, or related field (60% above) Education (if blank, degree and/or field of study not specified) Degrees/Field of Study required: Bachelor of Engineering, Bachelor of Technology Degrees/Field of Study preferred: Certifications (if blank, certifications not specified)

Required Skills