Data Engineer - Databricks

PwC; Hyderabad, Pune District, Bengaluru; Hybrid; INR 1,200,000 - 2,400,000; Full time

Job summary

PwC in Hyderabad is seeking a data engineering specialist to design and develop robust ETL/ELT pipelines using Databricks and PySpark.

The role focuses on Delta Lake architectures, DABs, SQL queries, data quality, and collaboration with data scientists and analysts to meet data requirements and optimize performance.

Responsibilities

• Design and develop robust ETL/ELT pipelines using Databricks platform and PySpark
• Implement and maintain data workflows using Databricks Asset Bundles (DABs)
• Build and optimize data processing solutions using Delta Lake architecture
• Develop complex SQL queries and stored procedures for data transformation and analysis
• Ensure data quality through comprehensive testing and validation frameworks
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements
• Monitor and optimize pipeline performance and data processing efficiency

Job description

Key Responsibilities

• Design and develop robust ETL/ELT pipelines using Databricks platform and PySpark
• Implement and maintain data workflows using Databricks Asset Bundles (DABs)
• Build and optimize data processing solutions using Delta Lake architecture
• Develop complex SQL queries and stored procedures for data transformation and analysis
• Ensure data quality through comprehensive testing and validation frameworks
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements
• Monitor and optimize pipeline performance and data processing efficiency