Data Engineer-Pyspark

Deloitte US-India Offices; Pune District; On-site; INR 2,000,000 - 4,000,000; Full time

Job summary

Deloitte US-India Offices in Pune is seeking an experienced Data Engineer to design, develop, test, and maintain large-scale data pipelines using PySpark. You will collaborate with cross-functional teams, implement ETL processes, and ensure scalability, reliability, and data security across a centralized data warehouse.

The ideal candidate has 6–11 years in data engineering, strong SQL and Hive/Hadoop experience, and a BE/BTech in any specialization.

Qualifications

• 6-11 years of experience in Data Engineering with expertise in PySpark.
• Strong understanding of Database Development, Data Warehousing Concepts & Tools (e.g., SQL), Big Data Technologies (Hadoop ecosystem).
• Experience working on big data projects involving Hive queries writing.

Responsibilities

• Design, develop, test, deploy, and maintain large-scale data pipelines using PySpark.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.
• Develop complex ETL processes to extract, transform, and load data from various sources into a centralized warehouse.
• Ensure scalability, reliability, and performance of the data pipeline by monitoring its components.
• Troubleshoot issues related to data quality, integrity, and security.

Skills

Team collaboration

Problem solving

Education

Bachelor's degree (B.Tech / B.E.)

Tools

PySpark

Hive

SQL

Hadoop

Job description

Roles and Responsibilities

• Design, develop, test, deploy, and maintain large-scale data pipelines using PySpark.
  
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.
  
• Develop complex ETL processes to extract, transform, and load data from various sources into a centralized warehouse.
  
• Ensure scalability, reliability, and performance of the data pipeline by monitoring its components.
  
• Troubleshoot issues related to data quality, integrity, and security.
  

Desired Candidate Profile

• 6-11 years of experience in Data Engineering with expertise in PySpark.
  
• Strong understanding of Database Development, Data Warehousing Concepts & Tools (e.g., SQL), Big Data Technologies (Hadoop ecosystem).
  
• Experience working on big data projects involving Hive queries writing.
  
• Bachelor's degree (B.Tech / B.E.) in Any Specialization.