GCP Data Engineer

Tata Consultancy Services; Tamil Nadu; On-site; INR 1,500,000 - 3,200,000; Full time

Job summary

Tata Consultancy Services in Tamil Nadu is looking for a seasoned GCP Data Engineer to design, build, and maintain scalable data pipelines on Google Cloud Platform, including BigQuery, GCS, Pub/Sub, and DBT. You will implement data models, quality checks, and monitoring to deliver reliable analytics solutions across the enterprise.

The role requires 5+ years in data engineering, strong SQL skills, and hands-on experience with Cloud Composer (Airflow), data warehousing concepts, and

Qualifications

• 5+ years of experience in Data Engineering.
• Minimum 3+ years of hands-on experience in GCP Data Engineering solutions.
• Strong experience with BigQuery, GCS, Pub/Sub, Cloud Composer (Airflow), and DBT.
• Experience in implementing monitoring, alerting, and data quality frameworks.
• Hands-on experience in designing and supporting production-grade data pipelines.
• Knowledge of security, governance, and best practices across data platforms.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines on GCP.
• Build data ingestion frameworks using Pub/Sub, GCS, and other GCP services.
• Develop and manage workflows using Cloud Composer (Apache Airflow).
• Create and optimize data models and transformations using DBT.
• Implement robust data quality checks, validation rules, and data quality scans.
• Design and maintain enterprise-scale data warehouses in BigQuery.
• Monitor production pipelines and implement proactive monitoring and alerting mechanisms.
• Troubleshoot and resolve data pipeline, performance, and quality issues.
• Collaborate with data architects, analysts, and business stakeholders to deliver data solutions.
• Ensure adherence to security, governance, and best practices across data platforms.
• Optimize query performance and cloud infrastructure costs.

Skills

BigQuery

Pub/Sub

Cloud Composer (Airflow)

Monitoring & Alerting

ETL/ELT Development

Data Pipeline Design and Development

Data Warehousing Concepts

Advanced SQL

Python

SQL

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field

Tools

DBT

Terraform

GitHub/GitLab

Spark/PySpark

Dataflow

Job description

We are seeking a skilled GCP Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable data pipelines and cloud-based data solutions on Google Cloud Platform (GCP). The ideal candidate should have strong expertise in BigQuery, GCS, Pub/Sub, Cloud Composer (Airflow), DBT, data quality frameworks, and monitoring solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines on GCP.
• Build and optimize data ingestion frameworks using Pub/Sub, GCS, and other GCP services.
• Develop and manage workflows using Cloud Composer (Apache Airflow).
• Create and optimize data models and transformations using DBT.
• Implement robust data quality checks, validation rules, and data quality scans.
• Design and maintain enterprise-scale data warehouses in BigQuery.
• Monitor production pipelines and implement proactive monitoring and alerting mechanisms.
• Troubleshoot and resolve data pipeline, performance, and quality issues.
• Collaborate with data architects, analysts, and business stakeholders to deliver data solutions.
• Ensure adherence to security, governance, and best practices across data platforms.
• Optimize query performance and cloud infrastructure costs.

Mandatory Skills

• BigQuery
• Pub/Sub
• Cloud Composer (Apache Airflow)
• Monitoring & Alerting
• ETL/ELT Development
• Data Pipeline Design and Development
• Data Warehousing Concepts
• Advanced SQL

Data Quality & Governance

• Data Quality Scans
• Data Validation Frameworks
• Data Monitoring and Observability

Data Transformation

• DBT (Data Build Tool)
• Data Modeling
• Performance Optimization

Programming

• Python
• SQL

Required Experience

• 5+ years of experience in Data Engineering.
• Minimum 3+ years of hands-on experience in GCP Data Engineering solutions.
• Strong experience with BigQuery, GCS, Pub/Sub, Cloud Composer (Airflow), and DBT.
• Experience in implementing monitoring, alerting, and data quality frameworks.
• Hands-on experience in designing and supporting production-grade data pipelines.

Preferred Skills

• Dataflow
• Terraform
• GitHub/GitLab
• Spark/PySpark

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.