Databricks Engineer

PwC India; Mumbai; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

PwC India seeks a Databricks engineer to lead migration of on-prem data to cloud Databricks environments, build scalable data pipelines, and optimize Spark jobs. You will collaborate with data scientists and analysts to deliver analytics solutions and ensure data quality and governance across ecosystems.

The role emphasizes practical experience with Databricks stacks, Unity Catalog, Delta Tables, and DLT, with strong SQL skills and knowledge of BI/data warehousing concepts.

Qualifications

• 4+ years of professional Data & Analytics experience with a global org.
• 4-8 years of Databricks experience.
• Leadership in Data & Analytics product development lifecycle.
• ETL development with Databricks & Spark.
• Data processing pipelines using Databricks, APIs & cloud services.
• Experience with Unity Catalog, Delta Tables, and DLT.
• Strong SQL writing & optimization.
• Domain knowledge in BI & data warehousing.

Responsibilities

• Migrate on-prem data sources to cloud using Databricks architecture.
• Develop and maintain data pipelines and analytics solutions in a cloud environment.
• Collaborate with data scientists, analysts, and stakeholders to define data requirements.
• Automate data workflows and schedule jobs using Databricks Workflows or orchestrators.
• Ensure data quality, governance, and lineage across pipelines.
• Perform code reviews, performance tuning, and monitoring of Spark jobs.
• Document architectures, configurations, and processes.

Skills

Databricks

PySpark

SQL

ETL/ELT

Data warehousing

Data pipelines

Spark optimization

Cloud data

Code reviews

Unit testing

Git

Workflow automation

Tools

Databricks

Apache Spark

PySpark

Airflow

Azure Data Factory

AWS S3

Job description

Databricks engineer Roles and Responsibilities:

Responsible for data management activities related to the migration of on-prem sources to cloud systems using Databricks architecture and solutions. In this role, you will be responsible for the development and maintenance of data pipelines and analytics solutions in a cloud-based environment.

Desired Candidate Profile:

• Minimum 4 years of professional experience with working knowledge in a Data and Analytics role with a Global organization
• 4 to 8 years of experience in working with Databricks tech stacks
• Experience in leading development of Data and Analytics products, from Requirement Gathering State to Driving User Adoption
• Develop and optimize ETL processes using Databricks and related tools like Apache Spark
• Design efficient data processing systems and pipelines using Databricks, APIs, and other cloud services
• Candidate with strong data transformation experience on Unity Catalog, Delta Tables, DLT
• Strong proficiency in writing and optimizing SQL queries and working with databases
• Ability to acquire specialized domain knowledge required to be more effective in all work activities
• BI & Data-warehousing concepts are a must.
• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark on Databricks.
• Ingest and transform data from multiple structured and unstructured sources including cloud storage (Azure Data Lake, AWS S3, etc.).
• Optimize Spark jobs for performance and cost-efficiency on the Databricks platform.
• Collaborate with data scientists, analysts, and stakeholders to understand data requirements and deliver high-quality solutions.
• Implement best practices in data engineering, including modular coding, unit testing, and version control (e.g., Git).
• Automate data workflows and schedule jobs using Databricks Workflows or external orchestration tools (e.g., Airflow, Azure Data Factory).
• Ensure data quality, integrity, and governance in all data pipelines.
• Participate in code reviews, performance tuning, and system monitoring.
• Document solutions, processes, and configurations.