Gcp Data Engineer

3shool Technology Consultants; Chennai District; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

3shool Technology Consultants in Chennai, India, is seeking a data engineering specialist to design and implement scalable ETL/ELT pipelines on Google Cloud Platform. You will build and optimize BigQuery data warehouses, orchestrate Airflow workflows via Cloud Composer, and deploy Google Cloud Functions integrated with multiple GCP services.

Python-based automation and strong SQL are essential. Responsibilities include ensuring performance, reliability, security, and cost optimization of data

Qualifications

• Experience with Google Cloud services and data engineering.
• Strong SQL skills for large-scale datasets.
• Proficient in Python with production-grade code.
• Experience with ETL/ELT pipelines and data warehousing.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using GCP services.
• Build and optimize data warehouse solutions using BigQuery.
• Develop and manage complex workflows using Cloud Composer (Apache Airflow).
• Develop and deploy Google Cloud Functions integrated with various GCP services.
• Build automated data processing frameworks using Python.
• Ensure performance, reliability, security, and cost optimization of data pipelines.

Skills

SQL

Python

REST APIs

Tools

Google BigQuery

Cloud Composer

Apache Airflow

Google Cloud Functions

GCS

Pub/Sub

Cloud Scheduler

Secret Manager

IAM

Cloud Logging

Cloud Monitoring

Git

CI/CD

DevOps practices

Parquet

Avro

JSON

CSV

Job description

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using GCP services.
• Build and optimize data warehouse solutions using BigQuery.
• Develop and manage complex workflows using Cloud Composer (Apache Airflow).
• Develop and deploy Google Cloud Functions integrated with various GCP services.
• Build automated data processing frameworks using Python.
• Ensure performance, reliability, security, and cost optimization of data pipelines.

Required Technical Skills

• Strong expertise in Google BigQuery data warehousing, partitioning, clustering, performance tuning, query and cost optimization.
• Hands-on experience with Cloud Composer / Apache Airflow.
• Strong Python programming skills with production-grade, reusable code.
• Experience with Google Cloud Functions and GCP integrations.
• Good knowledge of:
  • Google Cloud Storage (GCS)
  • Pub/Sub
  • Cloud Scheduler
  • Secret Manager
  • IAM
  • Cloud Logging
  • Cloud Monitoring
• Strong SQL skills and experience handling large-scale datasets.
• Experience with REST APIs, JSON, CSV, Parquet, and Avro.
• Familiarity with Git, CI/CD, and DevOps best practices.