GCP Data Engineer

Tata Consultancy Services; Delhi; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services is seeking a data engineer to design and build scalable data pipelines using Google Cloud Platform services. You will work with data engineers, analysts, and data scientists to deliver reliable data solutions for analytical workloads.

The role emphasizes ETL/ELT design, data modeling, and implementing CI/CD and IaC using Terraform, with emphasis on security and performance optimization. Experience with BigQuery, Dataflow, Pub/Sub and Cloud Composer is preferred.

Qualifications

• Strong experience in ETL/ELT design and data integration.
• Proficiency in Python and Java/Scala for data engineering.
• Experience with GCP data services (BigQuery, Dataflow, Cloud Composer).
• Familiarity with data modeling, warehousing, and streaming data.

Responsibilities

• Design and build scalable data pipelines using GCP services.
• Implement CI/CD and IaC with Terraform.
• Monitor, optimize, and troubleshoot data workloads and systems.
• Collaborate with data scientists and analysts to meet data needs.
• Ensure data security, quality, and compliance.

Skills

Python

Java/Scala

SQL

GCP

ETL/ELT

Data Modeling

Data Warehousing

Tools

BigQuery

Dataflow

Cloud Composer

Cloud Spanner

GCS

DBT

Airflow

Kafka

Jenkins

GitHub

Terraform

Dataproc

Pub/Sub

Job description

- Proficiency in programming languages: Python, Java

- Active experience on GCP tools and technologies like Big Query, Dataflow, Cloud Composer, Cloud Spanner, GCS, DBT etc.,

- Experience in ETL (Extract, Transform, Load) processes

- Knowledge of DevOps tools like Jenkins, GitHub, Terraform is desirable. Should have good knowledge on Kafka (Batch/ streaming)

- Understanding of Data models and experience in performing ETL design and build, database replication using Message based CDC

- Familiarity with cloud storage solutions

- Understanding of data security and scalability

• - Proficiency in relevant tools like Apache Airflow

Core Skills:

• Programming: SQL, Python, Java/Scala (for Dataflow/Spark).
• Data Concepts: Data Warehousing, ETL, Data Modeling, Streaming.

Key Responsibilities*

• Design & Build Data Pipelines: Develop scalable data ingestion, transformation (ETL/ELT), and orchestration workflows using GCP services.
• GCP Service Utilization: Implement solutions with BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Cloud Composer.
• Data Integration: Connect and manage data from diverse sources (APIs, databases, files).
• Optimization: Improve performance, cost-efficiency, and reliability of data processes.
• Collaboration: Work with data scientists, analysts, and architects to meet data needs.
• Quality & Security: Ensure data integrity, security, and compliance.
• Automation: Implement CI/CD and Infrastructure as Code (IaC) using tools like Terraform.
• Monitoring & Support: Maintain, monitor, troubleshoot data workloads and systems.