Senior Consultant - Cloud Data Engineer

enGen Global; Chennai District; On-site; INR 3,500,000 - 7,000,000; Full time

Job summary

enGen Global in Chennai, India seeks a Lead PySpark Cloud Data Engineer to design and implement scalable data pipelines on GCP, leading a team of engineers across locations. You will drive ingestion, transformation, and governance for enterprise data assets, mentoring juniors and ensuring production-grade quality.

Key responsibilities include building pipelines with PySpark and Dataproc, utilizing Iceberg and BigLake Metastore, and delivering production-ready data models with dbt.

Qualifications

• Bachelor's degree or higher from a reputed university.
• 8 to 12 years total experience with majority of that experience related to building high performant data pipelines batch and streaming.
• Strong hands-on experience in PySpark for data processing, transformation, and analysis.
• Extensive experience in implementing large scale data ingestion and curation solutions.
• Hands-on experience with cloud-based data platforms, preferably Google Cloud Platform (GCP) and services like Dataproc, BigQuery, and Cloud Storage.
• Knowledge in Apache Iceberg for open table formats.
• Advanced SQL skills for data querying, manipulation, and optimization.
• Proficiency with Git and collaborative development workflows.
• Excellent analytical and problem-solving skills with a keen attention to detail.
• Strong communication and interpersonal skills, with the ability to explain complex technical concepts to both technical and non-technical audiences.

Responsibilities

• Design, develop, and implement highly scalable, reliable, and performant data pipelines using PySpark, Dataproc, and other Google cloud-native technologies.
• Create pipelines for data lakehouse architectures utilizing Apache Iceberg for open table formats and BigLake Metastore for unified metadata management.
• Develop and maintain data transformation logic using PySpark and/or dbt to create clean, consistent, and production-ready data models.
• Work extensively with Google Cloud Platform (GCP) services, including Dataproc, BigQuery, Cloud Storage, and other relevant data services.
• Optimize data pipelines and queries for performance, efficiency, and cost-effectiveness.
• Implement and enforce data quality checks, monitoring, and governance best practices to ensure data integrity and reliability.
• Provide technical guidance and mentorship to junior data engineers, fostering a culture of continuous learning and improvement.
• Create and maintain comprehensive technical documentation for data pipelines, data models, and platform architecture.
• Provide regular updates on the tasks, status and risks to project manager.

Skills

PySpark

SQL

Git

GCP

BigQuery

Dataproc

Education

Bachelor's degree

Tools

Apache Iceberg

BigLake Metastore

dbt

Kafka

Pub/Sub

Atlan

Monte Carlo

Job description

Job Summary

Lead PySpark Cloud Data Engineer: This job involves understand the overall requirement of the enterprise Data need and design develop the robust, highly scalable resilient data pipeline using PySpark, Dataproc, open source table formats and other Google services. Job will involve extensive interfacing co-ordination with other senior tech folks (across India US) and lead the design development of the data ingestion pipelines

Responsibilities

• Design, develop, and implement highly scalable, reliable, and performant data pipelines using PySpark, Dataproc, and other Google cloud-native technologies


• Create pipelines for data lakehouse architectures utilizing Apache Iceberg for open table formats and BigLake Metastore for unified metadata management


• Develop and maintain data transformation logic using Pyspark and / or dbt to create clean, consistent, and production-ready data models


• Work extensively with Google Cloud Platform (GCP) services, including Dataproc, BigQuery, Cloud Storage, and other relevant data services


• Optimize data pipelines and queries for performance, efficiency, and cost-effectiveness


• Implement and enforce data quality checks, monitoring, and goverce best practices to ensure data integrity and reliability


• Provide technical guidance and mentorship to junior data engineers, fostering a culture of continuous learning and improvement


• Create and maintain comprehensive technical documentation for data pipelines, data models, and platform architecture


• Provide regular updates on the tasks, status and risks to project manager

Requirements

Required

• Bachelors degree or higher from a reputed university


• 8 to 12 years total experience with majority of that experience related to building high performant data pipelines batch and streaming


• Strong Hands on experience / expert-level proficiency in PySpark for data processing, transformation, and analysis


• Extensive experience in implementing large scale data ingestion and curation solutions


• Hands-on experience with cloud-based data platforms, preferably Google Cloud Platform (GCP) and services like Dataproc, BigQuery, and Cloud Storage


• Knowledge in Apache Iceberg for open table formats


• Advanced SQL skills for data querying, manipulation, and optimization


• Proficiency with Git and collaborative development workflows


• Excellent analytical and problem-solving skills with a keen attention to detail


• Strong communication and interpersonal skills, with the ability to explain complex technical concepts to both technical and non-technical audiences

Good to have

• Expertise in Google Cloud services


• Proven experience with Apache Iceberg for open table formats and BigLake Metastore for unified metadata management


• Strong experience with dbt (data build tool) for data modeling, transformation, and orchestration


• Experience in Data Goverce and Data quality tools like Atlan, Monte Carlo etc.


• Experience in processing streaming data using Kafka / Pub-Sub


• Healthcare industry experience


• Experience in Agile