Gcp Data Engineer

Tekskills; Pune District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Bootstrap a collaborative, independent role with strong debugging skills and a focus on performance tuning and automation within CI/CD pipelines.

Qualifications

• Strong SQL/PL-SQL querying and data manipulation.
• Proficiency in Python for data processing and automation.
• Hands-on experience with Google Cloud Platform services: BigQuery, Cloud Storage, Cloud Logging, Dataproc, Pub/Sub.
• Experience with GitHub and CI/CD pipelines for automation and deployment.
• Performance tuning and testing of ELT processes.
• Good analytical and debugging skills; able to resolve data issues.

Responsibilities

• Develop, implement, and optimize ETL/ELT pipelines for processing large datasets efficiently.
• Work extensively with BigQuery for data processing, querying, and optimization.
• Utilize Cloud Storage, Cloud Logging, Dataproc, and Pub/Sub for ingestion, storage, and event-driven processing.
• Perform performance tuning and testing of the ELT platform for high efficiency and scalability.
• Debug issues, perform root-cause analysis, and provide production incident solutions.
• Ensure data quality, accuracy, and integrity across pipelines.
• Collaborate with cross-functional teams to define tech requirements and deliver solutions.
• Work independently on assigned tasks with high productivity.

Skills

SQL

PL/SQL

Python

Data modeling

Data warehousing

Debugging

Analytical thinking

Independence

Tools

BigQuery

Cloud Storage

Cloud Logging

Dataproc

Pub/Sub

GitHub

CI/CD

Job description

Role & responsibilities

• Develop, implement, and optimize ETL/ELT pipelines for processing large datasets efficiently.
• Work extensively with BigQuery for data processing, querying, and optimization.
• Utilize Cloud Storage, Cloud Logging, Dataproc, and Pub/Sub for data ingestion, storage, and event-driven processing.
• Perform performance tuning and testing of the ELT platform to ensure high efficiency and scalability.
• Debug technical issues, perform root cause analysis, and provide solutions for production incidents.
• Ensure data quality, accuracy, and integrity across data pipelines.
• Collaborate with cross-functional teams to define technical requirements and deliver solutions.
• Work independently on assigned tasks while maintaining high levels of productivity and efficiency.

Skills Required:

• Proficiency in SQL and PL/SQL for querying and manipulating data.
• Experience in Python for data processing and automation.
• Hands-on experience with Google Cloud Platform (GCP), particularly:
  • BigQuery (must-have)
  • Cloud Storage
  • Cloud Logging
  • Dataproc
  • Pub/Sub
• Experience with GitHub and CI/CD pipelines for automation and deployment.
• Performance tuning and performance testing of ELT processes.
• Strong analytical and debugging skills to resolve data and pipeline issues efficiently.
• Self-motivated and able to work independently as an individual contributor.
• Good understanding of data modeling, database design, and data warehousing concepts.

Preferred candidate profile