DT - Lead Engineer

Prodapt Solutions; Chennai; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Prodapt Solutions is seeking a skilled GCP Data Engineer to design, build, and optimize scalable data pipelines on Google Cloud Platform. You will work with BigQuery, Dataproc, and Cloud Composer to enable reliable data processing for cross-functional stakeholders.

You'll write efficient SQL, troubleshoot pipelines, ensure data governance and security, and collaborate with data analysts and scientists across global teams. A strong problem-solver with good communication will thrive in this role.

Qualifications

• Experience designing scalable data pipelines on GCP.
• Hands-on with BigQuery performance tuning and data warehousing.
• Experience with Dataproc (Spark/Hadoop) on GCP.
• SQL development and optimization for data transformation.
• DAG design and scheduling with Cloud Composer (Airflow).
• Troubleshooting data pipelines and data quality issues in production.
• Collaboration with analysts/scientists to meet data requirements.
• Knowledge of data governance, security and compliance on GCP.
• Code reviews and knowledge sharing within the team.
• Documentation of data flows and runbooks.

Responsibilities

• Design, develop, and maintain robust, scalable data pipelines on Google Cloud Platform (GCP).
• Work extensively with BigQuery (BQ) for data warehousing, query optimization, and large-scale data analysis.
• Build and manage big data processing workflows using Dataproc (Hadoop/Spark on GCP).
• Write, optimize, and troubleshoot complex SQL queries for data transformation, validation, and reporting.
• Design, schedule, and monitor data workflows using Cloud Composer (Airflow).
• Perform root-cause analysis and troubleshooting of pipeline failures, data quality issues, and performance bottlenecks in production environments.
• Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and deliver reliable data solutions.
• Ensure data governance, security, and compliance standards are followed across all GCP data engineering solutions.
• Participate in code reviews, design discussions, and knowledge-sharing sessions within the team.
• Document technical designs, data flows, and operational runbooks for supportability.

Skills

Communication skills

Analytical skills

Problem solving

Ownership mindset

Stakeholder management

Working under pressure

Cross-functional collaboration

Education

Bachelor's/Master's degree in Computer Science/ IT / Engineering

GCP certification: Professional Data Engineer

Tools

Dataproc

BigQuery

Cloud Composer

Docker

Kubernetes

Terraform

Git

Job description

Overview

We are looking for a skilled and detail-oriented GCP Data Engineer to join our growing data engineering team. The ideal candidate will have hands‑on experience designing, building, and optimizing scalable data pipelines on Google Cloud Platform, along with strong problem-solving and communication skills to collaborate effectively with cross‑functional and global stakeholders.

Key Responsibilities

• Design, develop, and maintain robust, scalable, and efficient data pipelines on Google Cloud Platform (GCP).
• Work extensively with BigQuery (BQ) for data warehousing, query optimization, and large‑scale data analysis.
• Build and manage big data processing workflows using Dataproc (Hadoop/Spark ecosystem on GCP).
• Write, optimize, and troubleshoot complex SQL queries for data transformation, validation, and reporting.
• Design, schedule, and monitor data workflows using Cloud Composer (Apache Airflow).
• Perform root‑cause analysis and troubleshooting of pipeline failures, data quality issues, and performance bottlenecks in production environments.
• Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and deliver reliable data solutions.
• Ensure data governance, security, and compliance standards are followed across all GCP data engineering solutions.
• Participate in code reviews, design discussions, and knowledge‑sharing sessions within the team.
• Document technical designs, data flows, and operational runbooks for supportability.

Requirements

Required Technical Skills

Skill Proficiency Expected Google Cloud Platform (GCP) Strong hands‑on experience BigQuery (BQ) Query optimization, partitioning, clustering, cost management Dataproc Spark/Hadoop job design and management SQL Advanced query writing, performance tuning Cloud Composer / Apache Airflow DAG design, scheduling, monitoring Troubleshooting Strong analytical and debugging skills for pipeline/data issues

Good to Have (Additional Skills)

• Ab Initio - ETL development and pipeline migration experience.
• DevOps - CI/CD pipelines, version control (Git), containerization (Docker/Kubernetes), infrastructure as code (Terraform) is a plus.
• Prior Data/ETL Migration Experience - Hands‑on experience migrating data pipelines, ETL workloads, or on‑premise data warehouses to cloud platforms (e.g., on‑prem to GCP, or cross‑cloud migrations) is a strong value add.

Soft Skills

• Excellent communication skills - ability to clearly articulate technical concepts to both technical and non‑technical stakeholders.
• Strong ownership mindset and ability to work independently as well as within a team.
• Good stakeholder management skills, especially in a global/MNC delivery environment.
• Ability to work under pressure and manage multiple priorities in a fast‑paced environment.

Qualifications

• Bachelor's/Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• GCP certification (e.g., Professional Data Engineer) is a plus.