GCP Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a data engineer to design and build scalable data pipelines on the Google Cloud Platform. You will architect ingestion, transformation, and orchestration using BigQuery, Dataflow, Dataproc, Cloud Composer, and Airflow, while collaborating with data scientists and analysts to meet data needs.

The role emphasizes data modeling, ETL/ELT design, and secure, scalable architectures with CI/CD and IaC using Terraform.

Qualifications

• Proficiency in Python and Java/Scala for data processing and scripting.
• Active experience with GCP tools and technologies (BigQuery, Dataflow, Cloud Composer).
• Experience in ETL/ELT design, data modeling, and data integration tasks.
• Knowledge of DevOps tools (Jenkins, GitHub, Terraform) and Kafka (batch/streaming).
• Understanding of data models, data security, and scalable architectures.
• Familiarity with cloud storage solutions and Apache Airflow.

Responsibilities

• Design and build scalable data pipelines including ingestion, transformation, and orchestration using GCP services.
• Utilize BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Cloud Composer for data workflows.
• Connect and manage data from diverse sources (APIs, databases, files).
• Improve performance, cost-efficiency, and reliability of data processes.
• Collaborate with data scientists, analysts, and architects to meet data needs.
• Ensure data integrity, security, and compliance across systems.
• Implement CI/CD and Infrastructure as Code (IaC) using Terraform.
• Monitor, troubleshoot, and support data workloads and systems.

Skills

SQL

Python

Java/Scala

Data Warehousing

ETL/ELT

Streaming Data

Tools

BigQuery

Dataflow

Cloud Composer

Cloud Spanner

GCS

DBT

Apache Airflow

Jenkins

GitHub

Terraform

Kafka

Job description

- Proficiency in programming languages: Python, Java

- Active experience on GCP tools and technologies like Big Query, Dataflow, Cloud Composer, Cloud Spanner, GCS, DBT etc.,

- Experience in ETL (Extract, Transform, Load) processes

- Knowledge of DevOps tools like Jenkins, GitHub, Terraform is desirable. Should have good knowledge on Kafka (Batch/ streaming)

- Understanding of Data models and experience in performing ETL design and build, database replication using Message based CDC

- Familiarity with cloud storage solutions

- Understanding of data security and scalability

• - Proficiency in relevant tools like Apache Airflow

Core Skills:

• Programming: SQL, Python, Java/Scala (for Dataflow/Spark).
• Data Concepts: Data Warehousing, ETL, Data Modeling, Streaming.

Key Responsibilities*

• Design & Build Data Pipelines: Develop scalable data ingestion, transformation (ETL/ELT), and orchestration workflows using GCP services.
• GCP Service Utilization: Implement solutions with BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Cloud Composer.
• Data Integration: Connect and manage data from diverse sources (APIs, databases, files).
• Optimization: Improve performance, cost-efficiency, and reliability of data processes.
• Collaboration: Work with data scientists, analysts, and architects to meet data needs.
• Quality & Security: Ensure data integrity, security, and compliance.
• Automation: Implement CI/CD and Infrastructure as Code (IaC) using tools like Terraform.
• Monitoring & Support: Maintain, monitor, and troubleshoot data workloads and systems.