GCP Data Engineer

Zorba Consulting; Mumbai, Delhi, Bengaluru; Hybrid; INR 1,200,000 - 2,000,000; Full time

Job summary

Zorba Consulting seeks an experienced GCP Data Engineer with 5+ years in data engineering to design, implement, monitor, and optimize data ingestion pipelines on Google Cloud Platform across GCS, BigQuery, Dataflow, and related services.

You will develop ETL/ELT workflows, manage pipelines, configurations, and metadata, write Python scripts, and use Terraform and Git for infrastructure and version control in production environments.

Qualifications

• 5+ years of experience in data engineering or similar role.
• Strong hands-on experience with Google Cloud Platform (GCP).
• Hands-on experience with: GCS, BigQuery, Dataflow, Cloud Composer, Data Fusion, Dataproc.
• Strong Python skills for scripting, debugging, and automation.
• Experience with Terraform / IaC.
• Good understanding of Git and version control.
• Strong understanding of data ingestion, ETL/ELT workflows, and file formats.
• Ability to independently troubleshoot and resolve production/runtime data issues.

Responsibilities

• Configure and execute data ingestion pipelines using reusable GCP-based frameworks.
• Develop and manage ETL/ELT workflows using GCS, BigQuery, Dataflow, Composer, Data Fusion, and Dataproc.
• Manage ingestion parameters, job configurations, metadata, and runtime dependencies.
• Monitor daily data pipeline executions and ensure SLAs are met.
• Troubleshoot and resolve data, parameter, configuration, and job execution issues.
• Analyze runtime failures and independently identify and implement solutions.
• Escalate framework-level defects to the core development team when required.
• Support data engineering and migration projects on GCP.
• Develop scripts and automation using Python.
• Maintain infrastructure configurations and deployment scripts using Terraform.
• Follow Git-based version control and deployment best practices.
• Collaborate with cross-functional teams to ensure smooth delivery and production operations.
• Identify opportunities to improve ingestion frameworks, processes, and operational efficiency.
• Contribute to framework enhancements and automation initiatives where applicable.
• Work with TWS/TWSd or other enterprise scheduling tools for job scheduling and monitoring, where required.

Skills

Python

Terraform

Git

GCS

BigQuery

Dataflow

Cloud Composer

Data Fusion

Dataproc

Tools

Google Cloud Platform (GCP)

Job description

• We are looking for an experienced GCP Data Engineer with 5+ years of experience in data engineering and hands-on expertise in Google Cloud Platform.
• The candidate will be responsible for implementing, managing, monitoring, and troubleshooting data ingestion pipelines using GCP services.
• The role requires strong experience with GCS, BigQuery, Dataflow, Cloud Composer, Data Fusion, and Dataproc, along with proficiency in Python, Terraform, and Git.
• The ideal candidate should be comfortable working independently on data ingestion, ETL/ELT, migration, and production support activities.

Key Responsibilities

• Configure and execute data ingestion pipelines using reusable GCP-based frameworks.
• Develop and manage ETL/ELT workflows using GCS, BigQuery, Dataflow, Composer, Data Fusion, and Dataproc.
• Manage ingestion parameters, job configurations, metadata, and runtime dependencies.
• Monitor daily data pipeline executions and ensure agreed SLAs are met.
• Troubleshoot and resolve data, parameter, configuration, and job execution issues.
• Analyze runtime failures and independently identify and implement appropriate solutions.
• Escalate framework-level defects to the core development team when required.
• Support data engineering and migration projects on GCP.
• Develop scripts and automation using Python.
• Maintain infrastructure configurations and deployment scripts using Terraform.
• Follow Git-based version control and deployment best practices.
• Collaborate with cross-functional teams to ensure smooth delivery and production operations.
• Identify opportunities to improve ingestion frameworks, processes, and operational efficiency.
• Contribute to framework enhancements and automation initiatives where applicable.
• Work with TWS/TWSd or other enterprise scheduling tools for job scheduling and monitoring, where required.

Mandatory Skills

• 5+ years of experience as a Data Engineer or in a similar role.
• Strong hands-on experience with Google Cloud Platform (GCP).
• Hands-on experience with: Google Cloud Storage (GCS) BigQuery Dataflow Cloud Composer Cloud Data Fusion Dataproc Strong Python skills for scripting, debugging, and automation.
• Experience with Terraform / Infrastructure as Code (IaC).
• Good understanding of Git and version control.
• Strong understanding of data ingestion, ETL/ELT workflows, and file formats.
• Ability to independently troubleshoot and resolve production/runtime data issues.
• Strong analytical and problem-solving skills.

Good to Have

• Experience with TWS/TWSd or other enterprise job scheduling and monitoring tools.
• Understanding of BigQuery SQL optimization.
• Experience in GCP data migration projects.
• Exposure to reusable data ingestion frameworks and production data operations.
• Preferred Candidate Profile The ideal candidate should be a hands-on GCP Data Engineer who can independently manage data ingestion pipelines, troubleshoot production issues, work with multiple GCP data services, and support ETL/ELT operations.
• Strong Python, Terraform, and Git knowledge is expected, along with the ability to work effectively in a production environment.

Location - Pan India,Mumbai, Delhi / NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune