GCP Data Engineer

Tata Consultancy Services; Chennai District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services seeks an experienced GCP Data Engineer to design, develop, and maintain scalable data pipelines on Google Cloud Platform. You will work with BigQuery, Dataflow, Cloud Composer, Cloud Spanner, GCS, and DBT, building batch and streaming solutions using Kafka and Apache Beam.

You will implement ETL/ELT processes, ensure data quality and security, and collaborate with cross-functional teams to deliver enterprise data solutions in a fast-paced environment.

Qualifications

• Proficiency in Python and Java and strong SQL skills.
• Hands-on experience with GCP services including BigQuery, Dataflow, Cloud Composer, Cloud Spanner, GCS, DBT.
• Expertise in Apache Beam (Dataflow) and ETL development.
• Experience with Kafka for batch and streaming processing.
• Knowledge of database replication and message-based CDC.
• Understanding of data modeling, scalability, and security.
• Familiarity with Apache Airflow, Jenkins, GitHub, and Terraform is desirable.

Responsibilities

• Design and build scalable ETL/ELT pipelines on GCP.
• Develop and optimize data processing workflows using Dataflow and BigQuery.
• Implement batch and streaming solutions leveraging Kafka and Apache Beam.
• Ensure data quality, security, performance, and scalability across platforms.
• Collaborate with cross-functional teams to deliver enterprise data solutions.

Skills

Python

Java

SQL

Kafka

ETL concepts

Data modeling

Security

Batch & streaming

Tools

Google Cloud Platform

BigQuery

Dataflow

Cloud Composer

Cloud Spanner

GCS

DBT

Apache Beam

Airflow

Jenkins

GitHub

Terraform

Job description

We are looking for an experienced GCP Data Engineer with strong expertise in designing, developing, and maintaining scalable data pipelines on Google Cloud Platform (GCP). The ideal candidate should have hands-on experience with BigQuery, Dataflow, Cloud Composer, Cloud Spanner, GCS, DBT, Python, SQL, and ETL processes. Experience in building batch and streaming data solutions using Kafka and Apache Beam (Dataflow) is essential.

Required Skills

• Strong programming skills in Python and Java
• Hands-on experience with GCP services: BigQuery, Dataflow, Cloud Composer, Cloud Spanner, GCS, DBT
• Expertise in Apache Beam (Dataflow) and ETL development
• Strong knowledge of SQL and data engineering concepts
• Experience with Kafka (Batch & Streaming)
• Knowledge of database replication and Message-Based CDC
• Good understanding of data modeling, scalability, and security
• Experience with Apache Airflow
• Exposure to Jenkins, GitHub, and Terraform is desirable

Keywords (Minimum 3 Must Match)

• GCP BigQuery
• Python
• ETL / SQL
• Google Cloud

Good to Have

Key Responsibilities

• Design and build scalable ETL/ELT pipelines on GCP.
• Develop and optimize data processing workflows using Dataflow and BigQuery.
• Implement batch and streaming solutions leveraging Kafka and Apache Beam.
• Ensure data quality, security, performance, and scalability across platforms.
• Collaborate with cross-functional teams to deliver enterprise data solutions.