GCP Data Engineer

Tata Consultancy Services; Tamil Nadu; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

Tata Consultancy Services in India is seeking a GCP Data Engineer with 5+ years of experience to design, develop, and maintain scalable data pipelines and cloud-based data solutions on Google Cloud Platform. The role focuses on BigQuery, GCS, Pub/Sub, Cloud Composer, DBT, data quality, monitoring, and cost optimization, with collaboration across data architects, analysts, and business stakeholders.

Strong SQL and Python skills, data modeling, and governance are required to deliver

Qualifications

• Experience designing scalable data pipelines on cloud platforms.
• Hands-on with BigQuery, GCS, Pub/Sub and Airflow-based workflows.
• Experience implementing data quality checks and monitoring.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines on GCP.
• Build and optimize data ingestion using Pub/Sub and GCS.
• Develop and manage workflows with Cloud Composer (Airflow).
• Create and optimize data models using DBT.
• Implement data quality checks and monitoring frameworks.
• Design and maintain enterprise-scale data warehouses in BigQuery.
• Monitor pipelines, resolve performance and quality issues.
• Collaborate with data architects, analysts, and stakeholders.
• Ensure governance, security, and cost optimization across platforms.
• Optimize query performance and cloud infrastructure costs.

Skills

ETL/ELT Development

Data Modeling

SQL Proficiency

Python

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

GCP

BigQuery

GCS

Pub/Sub

Cloud Composer (Airflow)

DBT

Monitoring & Alerting

Job description

Job Summary

We are seeking a skilled GCP Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable data pipelines and cloud-based data solutions on Google Cloud Platform (GCP). The ideal candidate should have strong expertise in BigQuery, GCS, Pub/Sub, Cloud Composer (Airflow), DBT, data quality frameworks, and monitoring solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines on GCP.
• Build and optimize data ingestion frameworks using Pub/Sub, GCS, and other GCP services.
• Develop and manage workflows using Cloud Composer (Apache Airflow).
• Create and optimize data models and transformations using DBT.
• Implement robust data quality checks, validation rules, and data quality scans.
• Design and maintain enterprise-scale data warehouses in BigQuery.
• Monitor production pipelines and implement proactive monitoring and alerting mechanisms.
• Troubleshoot and resolve data pipeline, performance, and quality issues.
• Collaborate with data architects, analysts, and business stakeholders to deliver data solutions.
• Ensure adherence to security, governance, and best practices across data platforms.
• Optimize query performance and cloud infrastructure costs.

Mandatory Skills

Google Cloud Platform (GCP)

• Google Cloud Storage (GCS)
• BigQuery
• Pub/Sub
• Cloud Composer (Apache Airflow)
• Monitoring & Alerting

Data Engineering

• ETL/ELT Development
• Data Pipeline Design and Development
• Data Warehousing Concepts
• Advanced SQL

Data Quality & Governance

• Data Quality Scans
• Data Validation Frameworks
• Data Monitoring and Observability

Data Transformation

• DBT (Data Build Tool)
• Data Modeling
• Performance Optimization

Programming

• Python
• SQL

Required Experience

• 5+ years of experience in Data Engineering.
• Minimum 3+ years of hands-on experience in GCP Data Engineering solutions.
• Strong experience with BigQuery, GCS, Pub/Sub, Cloud Composer (Airflow), and DBT.
• Experience in implementing monitoring, alerting, and data quality frameworks.
• Hands-on experience in designing and supporting production-grade data pipelines.

Preferred Skills

• Dataflow
• Terraform
• CI/CD Pipelines
• GitHub/GitLab
• Spark/PySpark
• Kubernetes

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.