Senior GCP Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a senior GCP Data Engineer to design and implement large-scale data solutions on Google Cloud Platform. You will lead data pipelines, govern data, and collaborate with stakeholders across analytics, security, and ML teams.

The role demands hands-on expertise with BigQuery, Dataflow, Dataproc, Cloud Composer, and Terraform, plus leadership to mentor juniors and enforce best practices.

Qualifications

• Expert-level knowledge of GCP services including BigQuery, Dataflow, Dataproc, and Cloud Composer.
• Strong Python (Pandas, PySpark) and SQL for data wrangling and transformation.
• Experience in ETL/ELT design and data lakehouse architecture.
• Experience with CI/CD and IaC (GitHub Actions, Terraform).

Responsibilities

• Lead scalable data pipelines using GCP native tools (Dataflow, Composer, BigQuery, Dataproc).
• Architect data lakehouse/warehouse solutions for analytics and AI/ML requirements.
• Mentor junior data engineers and ensure high-quality code.
• Collaborate with data architects on strategy, security, and governance.
• Oversee data ingestion from diverse sources into GCP.
• Implement ETL/ELT processes and optimize performance.
• Apply infrastructure automation using Terraform and CI/CD.
• Define and enforce governance, lineage, and metadata standards.
• Monitor workloads, optimize BigQuery costs, and troubleshoot pipelines.
• Drive proof-of-concepts for modernizing enterprise data platforms.

Skills

GCP

BigQuery

Dataflow

Dataproc

Cloud Composer

Pub/Sub

Cloud Storage

Python

SQL

ETL/ELT design

Data lakehouse

CI/CD

Terraform

IaC

GKE

Cloud Run

Data governance

Security

Tools

GitHub Actions

Jenkins

Cloud Build

Docker

Kubernetes (GKE)

Job description

(in Yrs.) Proven experience in designing, architecting, and implementing large-scale data solutions on Google Cloud Platform (GCP). Expertise in data ingestion, transformation, orchestration, and analytics using native GCP services. Hands-on experience in data migration, modernization, and data lakehouse architecture. Strong experience leading data engineering teams, reviewing code, and ensuring best practices. Deep understanding of data governance, cost optimization, and security within GCP. Ability to work closely with data architects, business stakeholders, and cross-functional teams to define enterprise data strategies. Experience in end-to-end lifecycle of data engineering projects using Agile and DevOps frameworks.

Key words to search in resume Senior GCP Data Engineer, Google Cloud, BigQuery, Dataflow, Dataproc, Cloud Composer, Pub/Sub, Cloud Storage, Python, SQL, Data Lake, Data Warehouse, ETL, Airflow, Terraform, DevOps, GKE, Cloud Run, Architecture, Leadership, Data Modernization, CI/CD

Technical/Functional Skills

Must Have Skills

• Expert-level knowledge of Google Cloud Platform services including:
• BigQuery – advanced SQL, partitioning, optimization, and cost control
• Dataflow (Apache Beam) – large-scale batch and streaming data pipelines
• Dataproc – Spark and Hadoop-based distributed processing
• Cloud Composer (Airflow) – workflow orchestration, scheduling, and dependency management
• Pub/Sub – event-driven, real-time ingestion
• Cloud Storage (GCS) – data lake setup, lifecycle management, access control
• Strong proficiency in Python (Pandas, PySpark, Beam SDK) and SQL for data wrangling and transformation.
• Expertise in ETL/ELT design, data lake/data warehouse/lakehouse architecture.
• Experience with DevOps & CI/CD using tools such as GitHub Actions, Jenkins, or Cloud Build.
• Hands-on with Terraform or Deployment Manager for Infrastructure as Code (IaC).
• Solid understanding of GCP IAM, VPC, security policies, and encryption mechanisms.
• Familiarity with data quality, metadata, lineage, and cataloging tools (e.g., Data Catalog).
• Proficiency in cost governance, performance tuning, and resource monitoring (Cloud Monitoring/Logging).
• Knowledge of modern data frameworks and data integration between systems like Kafka, Snowflake, or Looker.

Secondary Skills

• Understanding of machine learning workflows using BigQuery ML, Vertex AI, or TensorFlow.
• Familiarity with Docker, Kubernetes (GKE), and Cloud Run for containerized workloads.
• Experience with data governance frameworks and compliance standards (GDPR, HIPAA).
• Working knowledge of BI/analytics tools such as Looker, Power BI, or Tableau.
• Strong documentation, mentoring, and stakeholder communication skills.
• Exposure to multi-cloud environments (AWS, Azure) for data integration.

Responsibilities

• Lead the design and implementation of scalable data pipelines using GCP native tools (Dataflow, Composer, BigQuery, Dataproc).
• Architect and develop data lakehouse and warehouse solutions that meet business analytics and AI/ML requirements.
• Manage and mentor junior data engineers, ensuring high-quality and reusable code.
• Collaborate with data architects to align on data strategy, security, and best practices.
• Oversee data ingestion from diverse sources (on-prem, APIs, Kafka, databases) into GCP.
• Implement ETL/ELT processes, optimize for performance, and ensure data quality and reliability.
• Apply infrastructure automation using Terraform and CI/CD for seamless deployments.
• Define and enforce governance, lineage, and metadata standards.
• Monitor workloads, optimize BigQuery costs, and troubleshoot pipeline performance issues.
• Drive innovation and proof-of-concepts for modernizing enterprise data platforms.
• Present technical designs and architecture recommendations to senior stakeholders.
• Collaborate across teams (security, analytics, ML) to deliver end-to-end data-driven solutions.
• Ensure all data systems comply with organizational security, compliance, and DR policies.