GCP Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services is seeking a Data Engineer to design and build scalable data pipelines on Google Cloud Platform. You will implement ingestion, transformation, and orchestration workflows using tools like BigQuery, Dataflow, and Cloud Composer.

You will collaborate with data scientists and analysts to ensure data quality, security, and cost efficiency while optimizing performance across data systems.

Qualifications

• Proficient in SQL, Python, and Java/Scala for data processing.
• Experience with Data Warehousing, ETL/ELT, and data modeling.
• Familiar with cloud-based data tooling and orchestration.
• Knowledge of data security and scalability considerations.

Responsibilities

• Design and build scalable data pipelines using GCP services.
• Implement data ingestion, transformation, and orchestration workflows.
• Collaborate with data scientists and analysts to meet data requirements.
• Ensure data quality, security, and cost-efficiency.

Skills

SQL

Python

Java/Scala

Data Warehousing

ETL/ELT

Streaming

Airflow

CI/CD

Tools

BigQuery

Dataflow

Cloud Composer

Cloud Spanner

GCS

DBT

Dataproc

Pub/Sub

Kafka

Jenkins

GitHub

Terraform

Apache Airflow

Job description

- Proficiency in programming languages: Python, Java

- Active experience on GCP tools and technologies like Big Query, Dataflow, Cloud Composer, Cloud Spanner, GCS, DBT etc.,

- Experience in ETL (Extract, Transform, Load) processes

- Knowledge of DevOps tools like Jenkins, GitHub, Terraform is desirable. Should have good knowledge on Kafka (Batch/ streaming)

- Understanding of Data models and experience in performing ETL design and build, database replication using Message based CDC

- Familiarity with cloud storage solutions

- Understanding of data security and scalability

• - Proficiency in relevant tools like Apache Airflow

Core Skills:

• Programming: SQL, Python, Java/Scala (for Dataflow/Spark).
• Data Concepts: Data Warehousing, ETL, Data Modeling, Streaming.

Key Responsibilities*

• Design & Build Data Pipelines: Develop scalable data ingestion, transformation (ETL/ELT), and orchestration workflows using GCP services.
• GCP Service Utilization: Implement solutions with BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Cloud Composer.
• Data Integration: Connect and manage data from diverse sources (APIs, databases, files).
• Optimization: Improve performance, cost-efficiency, and reliability of data processes.
• Collaboration: Work with data scientists, analysts, and architects to meet data needs.
• Quality & Security: Ensure data integrity, security, and compliance.
• Automation: Implement CI/CD and Infrastructure as Code (IaC) using tools like Terraform.
• Monitoring & Support: Maintain, monitor, and troubleshoot data workloads and systems.