GCP Data Engineer

HTC Global Services; Chennai District; On-site; INR 1,400,000 - 2,600,000; Full time

Job summary

HTC Global Services invites a skilled Data Engineer to design, build, and maintain scalable data pipelines on Google Cloud Platform. You will work with data analysts, data scientists, and business stakeholders to enable data-driven decisions and reliable data platforms.

The role emphasizes Python development, BigQuery data modeling, CI/CD practices, and data governance. You will implement ETL/ELT workflows with Dataflow, Airflow, and related GCP services while ensuring security and cost

Qualifications

• Strong hands-on experience with Python for data engineering and automation.
• Solid experience with Google Cloud Platform (GCP).
• Hands-on experience with BigQuery and SQL.
• Experience with Google Cloud Storage (GCS).
• Experience with Cloud Composer / Apache Airflow.
• Experience with Dataflow / Apache Beam.
• Understanding of ETL/ELT concepts and data pipeline architecture.
• Good knowledge of relational databases and SQL optimization.
• Experience working with REST APIs and data formats such as JSON, CSV, Parquet.
• Experience with Git and CI/CD tools.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and GCP services.
• Develop high-quality, reusable Python-based data processing solutions.
• Build and manage data pipelines using Dataflow, Cloud Composer, BigQuery, and Cloud Storage.
• Design data models and optimize queries and workloads in BigQuery.
• Develop data ingestion processes from APIs, databases, files, and other data sources.
• Implement data validation, quality checks, monitoring, and error-handling mechanisms.
• Optimize data pipelines for performance, scalability, reliability, and cost.
• Collaborate with stakeholders to translate data requirements into technical solutions.
• Implement CI/CD processes and follow DevOps practices for data engineering solutions.
• Troubleshoot production data pipeline issues and perform root-cause analysis.
• Ensure data security, governance, access control, and compliance requirements are followed.
• Prepare technical documentation for data pipelines, architecture, and operations.

Skills

Python for data engineering

Google Cloud Platform (GCP)

BigQuery

SQL optimization

ETL/ELT concepts

REST APIs

CI/CD for data pipelines

Data governance

Data security

Airflow

Tools

Google Cloud Dataflow

Cloud Composer

Cloud Storage

Apache Beam

Git

Terraform

Job description

We are looking for an experienced Data Engineer with strong Python and Google Cloud Platform (GCP) expertise to design, develop, and maintain scalable data pipelines and cloud-based data solutions. The candidate will work closely with data analysts, data scientists, and business stakeholders to build reliable data platforms and enable data-driven decision-making.

Key Responsibilities

Design, develop, and maintain scalable ETL/ELT data pipelines using Python and GCP services.

Develop high-quality, reusable, and efficient Python-based data processing solutions.

Build and manage data pipelines using Google Cloud Dataflow, Cloud Composer, BigQuery, and Cloud Storage.

Design data models and optimize queries and workloads in Google BigQuery.

Develop data ingestion processes from APIs, databases, files, and other data sources.

Implement data validation, quality checks, monitoring, and error-handling mechanisms.

Optimize data pipelines for performance, scalability, reliability, and cost.

Work with stakeholders to understand data requirements and translate them into technical solutions.

Implement CI/CD processes and follow DevOps best practices for data engineering solutions.

Troubleshoot production data pipeline issues and provide root-cause analysis.

Ensure data security, governance, access control, and compliance requirements are followed.

Prepare technical documentation for data pipelines, architecture, and operational processes.

Required Skills

Strong hands-on experience with Python for data engineering and automation.

Strong experience with Google Cloud Platform (GCP).

Hands-on experience with BigQuery and SQL.

Experience with Google Cloud Storage (GCS).

Experience with Cloud Composer / Apache Airflow.

Experience with Dataflow / Apache Beam.

Strong understanding of ETL/ELT concepts and data pipeline architecture.

Good knowledge of relational databases and SQL optimization.

Experience working with REST APIs and different data formats such as JSON, CSV, and Parquet.

Experience with Git and CI/CD tools.

Understanding of cloud security, IAM, and data governance.

Preferred Skills

Experience with Pub/Sub and event-driven data pipelines.

Knowledge of Cloud Functions / Cloud Run.

Experience with Dataproc / Spark.

Knowledge of data warehousing and dimensional modeling.

Experience with Terraform or other Infrastructure-as-Code tools.

Exposure to data visualization tools such as Looker or Power BI.

Experience working in Agile/Scrum environments