Data Engineer-Pyspark

Deloitte US-India Offices; Chennai District; Hybrid; INR 900,000 - 1,300,000; Full time

Job summary

Deloitte US-India Offices is seeking a seasoned Data Engineer with PySpark expertise to build scalable data pipelines in Chennai. You will design, develop, test, and maintain end-to-end ETL processes in a collaborative environment.

Ideal candidates have 6–11 years of data engineering experience, strong SQL and data warehousing knowledge, and hands-on experience with Hadoop ecosystem and Hive. Bachelor's in B.Tech/B.E. is required.

Qualifications

• 6-11 years of experience in Data Engineering with expertise in PySpark.
• Strong understanding of Database Development, Data Warehousing Concepts & Tools (e.g., SQL), Big Data Technologies (Hadoop ecosystem).
• Experience working on big data projects involving Hive queries writing.
• Bachelor's degree (B.Tech / B.E.) in Any Specialization.

Responsibilities

• Design, develop, test, deploy, and maintain large-scale data pipelines using PySpark.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.
• Develop complex ETL processes to extract, transform, and load data from various sources into a centralized warehouse.
• Ensure scalability, reliability, and performance of the data pipeline by monitoring its components.
• Troubleshoot issues related to data quality, integrity, and security.

Skills

PySpark

SQL

Data Warehousing

Big Data Technologies

Education

Bachelor's degree (B.Tech / B.E.)

Tools

Hive

Hadoop ecosystem

Job description

Roles and Responsibilities

• Design, develop, test, deploy, and maintain large-scale data pipelines using PySpark.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.
• Develop complex ETL processes to extract, transform, and load data from various sources into a centralized warehouse.
• Ensure scalability, reliability, and performance of the data pipeline by monitoring its components.
• Troubleshoot issues related to data quality, integrity, and security.

Desired Candidate Profile

• 6-11 years of experience in Data Engineering with expertise in PySpark.
• Strong understanding of Database Development, Data Warehousing Concepts & Tools (e.g., SQL), Big Data Technologies (Hadoop ecosystem).
• Experience working on big data projects involving Hive queries writing.
• Bachelor's degree (B.Tech / B.E.) in Any Specialization.