GCP Data Engineer

ValueLabs; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Valuelabs in Hyderabad is seeking a hands-on Data Engineer to build, test, and run data pipelines on the Google Cloud Platform. You will collaborate with the Data Architect to deliver production-grade, cost-efficient pipelines feeding analytics, reporting and ML.

You will implement batch and streaming pipelines with Dataflow and Dataproc, optimize BigQuery datasets, and enforce data quality, security and governance.

Qualifications

• Hands-on experience with GCP data services and pipelines.
• Strong knowledge of SQL with performance tuning.
• Proficiency in Python and PySpark for data processing.
• Experience with data warehousing and dimensional modelling.
• Familiarity with Git and CI/CD tooling.
• Comfortable in Agile teams and fast-paced environments.

Responsibilities

• Build batch and streaming pipelines using Dataflow and Dataproc.
• Develop and optimise BigQuery datasets, tables, views and procedures.
• Ingest data from databases, APIs, files and SaaS sources into GCP.
• Orchestrate workflows with Cloud Composer and Airflow.
• Implement transformations with SQL, Python, dbt or Dataform.
• Build data-quality checks and monitor pipelines.
• Support migration of legacy ETL workloads to GCP.
• Write infrastructure as code with Terraform and deploy via CI/CD.
• Apply security and governance standards: IAM, PII masking, metadata.
• Troubleshoot issues and optimize performance to reduce cost.

Skills

BigQuery

Dataflow/Dataproc

Pub/Sub

SQL

Python

PySpark

ETL/ELT patterns

Git & CI/CD

Linux shell

REST APIs

Agile teamwork

Tools

Cloud Composer

Terraform

CI/CD (Cloud Build / GitHub Actions / Jenkins)

Looker Studio

Vertex AI

BigQuery ML

Job description

Job Description

Data Engineers build, test and run the data pipelines and models on our GCP platform. You will work with the Data Architect to turn designs into reliable, cost-efficient production pipelines that feed analytics, reporting and ML.

Key responsibilities

• Build batch and streaming pipelines using Dataflow (Apache Beam), Dataproc (PySpark), Pub/Sub and Datastream.
• Develop and optimise BigQuery datasets, tables, views and stored procedures; apply partitioning and clustering.
• Ingest data from databases, APIs, files and SaaS sources into Cloud Storage and BigQuery.
• Orchestrate and schedule workflows with Cloud Composer (Apache Airflow).
• Implement transformations and data models with SQL, Python, dbt or Dataform.
• Build data-quality checks, validation and reconciliation; monitor pipelines with Cloud Monitoring and Logging.
• Support migration of legacy ETL (Informatica, SSIS, Talend, Hadoop) workloads to GCP.
• Write infrastructure as code with Terraform and deploy through CI/CD pipelines.
• Apply security and governance standards: IAM roles, PII masking, Dataplex / Data Catalog metadata.
• Troubleshoot production issues, tune performance and reduce cost.
• Write technical documentation and take part in code reviews and Agile ceremonies.

Required skills

• Hands‑on experience with BigQuery, Dataflow or Dataproc, Pub/Sub, Cloud Storage and Cloud Composer.
• Strong SQL (complex joins, window functions, performance tuning).
• Strong Python; PySpark experience.
• Understanding of data warehousing, dimensional modelling and ETL / ELT patterns.
• Experience with Git and CI/CD (Cloud Build, GitHub Actions or Jenkins).
• Familiarity with Linux shell scripting and REST APIs.
• Good problem‑solving and communication skills; comfortable working in Agile teams.

Nice to have

• Java or Scala for Apache Beam.
• Experience with Kafka, Cloud Functions / Cloud Run, or Bigtable / Spanner / Cloud SQL.
• Exposure to Looker / Looker Studio, BigQuery ML or Vertex AI.
• Experience on AWS or Azure data services.