Data Engineer (PySpark + Cloudera)

Zorba AI; Maharashtra; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading AI technology company in Maharashtra is seeking an experienced Data Engineer to design, develop, and optimize scalable data pipelines. The ideal candidate should have over 6 years of experience in Data Engineering and strong expertise in PySpark and AWS services. Responsibilities include managing data workflows and implementing data governance principles. This role requires a deep understanding of modern data platforms and tools like Cloudera and Hive Metastore.

Qualifications

• 6+ years of experience in Data Engineering.
• Strong hands-on experience with PySpark.
• Deep understanding of modern data platforms and distributed data systems.
• Experience with Cloudera Data Platform (CDP) ecosystem.
• Proficiency in SQL and data modeling concepts.
• Experience with AWS data services (EMR, S3, MWAA, Glue, Lake Formation).
• Strong knowledge of Hive Metastore and big data architectures.
• Experience with file formats (Iceberg, Parquet) and optimization techniques.
• Familiarity with data governance, cataloging, and lineage tools (Atlan).

Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines using PySpark.
• Work with Cloudera Data Platform (CDP) components.
• Manage and optimize data workflows.
• Implement data governance, security, and access control.
• Develop and maintain data models and large-scale datasets.
• Collaborate with cross-functional teams to deliver data solutions.

Skills

PySpark

AWS

Cloudera

Job description

Job Summary

We are looking for experienced Data Engineers with strong expertise in PySpark and Cloudera Data Platform to design, develop, and optimize scalable data pipelines. The ideal candidate should have hands‑on experience with distributed data systems, cloud platforms (AWS), and modern data architecture, along with a strong understanding of data governance and cataloging tools.

Key Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines using PySpark
• Work with Cloudera Data Platform (CDP) components such as CDE, CDW, Ozone, and Airflow
• Manage and optimize data workflows, ensuring high performance and reliability
• Implement data governance, security, and access control using Apache Ranger
• Develop and maintain data models, Hive Metastore, and large-scale distributed datasets
• Collaborate with cross-functional teams to deliver data solutions for analytics and reporting
• Work with AWS services like EMR, S3, MWAA, Glue Catalog, and Lake Formation
• Ensure proper data partitioning, bucketing, and optimization using formats like Iceberg and Parquet
• Integrate data cataloging and lineage using Atlan

Required Skills & Qualifications

• 6+ years of experience in Data Engineering
• Strong hands‑on experience with PySpark
• Deep understanding of modern data platforms and distributed data systems
• Experience with Cloudera Data Platform (CDP) ecosystem
• Proficiency in SQL and data modeling concepts
• Experience with AWS data services (EMR, S3, MWAA, Glue, Lake Formation)
• Strong knowledge of Hive Metastore and big data architectures
• Experience with file formats (Iceberg, Parquet) and optimization techniques
• Familiarity with data governance, cataloging, and lineage tools (Atlan)

Skills: pyspark, aws, cloudera