GCP Data Engineer

Paltech Consulting; Hyderabad; Hybrid; INR 1,400,000 - 2,800,000; Full time

Job summary

Paltech Consulting is seeking a Cloud Data Engineer to design and optimize scalable data pipelines on Google Cloud Platform. You will build batch and streaming ingestion using Pub/Sub, Dataflow, Datastream, and Storage Transfer Service, load data into BigQuery, and manage GCS landing zones and storage policies.

You will develop ETL/ELT workflows with Dataflow, Dataproc, and Dataprep, implement medallion architecture, design cost-aware data warehouses, orchestrate with Cloud Composer, and enforce

Responsibilities

• Build batch and streaming data ingestion pipelines using Pub/Sub, Dataflow, Datastream (CDC), and Storage Transfer Service
• Load and ingest data into BigQuery via batch loads, the BigQuery Storage Write API and external/BigLake tables
• Manage Cloud Storage (GCS) landing zones, buckets, lifecycle policies and storage classes
• Develop ETL/ELT workflows using Dataflow (Apache Beam), Dataproc (Spark/Hadoop) and Dataprep
• Build SQL-based transformations using Dataform and BigQuery scheduled queries
• Implement medallion/layered architecture (raw staging curated/gold)
• Design and optimize BigQuery data warehouses with partitioning, clustering, materialized views and cost controls
• Orchestrate pipelines using Cloud Composer (Airflow) and Workflows
• Schedule and automate jobs with Cloud Scheduler and Cloud Functions
• Manage access and security using IAM, service accounts and Workload Identity Federation
• Implement data governance, lineage and quality using Knowledge Catalog
• Apply encryption (CMEK/CSEK, Cloud KMS), VPC Service Controls and DLP via Sensitive Data Protection
• Ensure compliance with PII masking and column/row-level security in BigQuery
• Enable analytics consumption through Looker, Looker Studio and BigQuery BI Engine
• Monitor pipelines and resources with Cloud Monitoring, Cloud Logging, and Cloud Trace
• Optimize query performance, slot usage, and overall cloud cost
• Implement CI/CD using Cloud Build, Artifact Registry, and Cloud Source Repositories
• Manage infrastructure-as-code using Terraform and Deployment Manager
• Maintain version control, automated testing and pipeline reliability with alerting and SLAs

Job description

Role & responsibilities

• Build batch and streaming data ingestion pipelines using Pub/Sub, Dataflow, Datastream (CDC), and Storage Transfer Service
• Load and ingest data into BigQuery via batch loads, the BigQuery Storage Write API and external/BigLake tables
• Manage Cloud Storage (GCS) landing zones, buckets, lifecycle policies and storage classes
• Develop ETL/ELT workflows using Dataflow (Apache Beam), Dataproc (Spark/Hadoop) and Dataprep
• Build SQL-based transformations using Dataform and BigQuery scheduled queries
• Implement medallion/layered architecture (raw staging curated/gold)
• Design and optimize BigQuery data warehouses with partitioning, clustering, materialized views and cost controls
• Orchestrate pipelines using Cloud Composer (Airflow) and Workflows
• Schedule and automate jobs with Cloud Scheduler and Cloud Functions
• Manage access and security using IAM, service accounts and Workload Identity Federation
• Implement data governance, lineage and quality using Knowledge Catalog
• Apply encryption (CMEK/CSEK, Cloud KMS), VPC Service Controls and DLP via Sensitive Data Protection
• Ensure compliance with PII masking and column/row-level security in BigQuery
• Enable analytics consumption through Looker, Looker Studio and BigQuery BI Engine
• Monitor pipelines and resources with Cloud Monitoring, Cloud Logging, and Cloud Trace
• Optimize query performance, slot usage, and overall cloud cost
• Implement CI/CD using Cloud Build, Artifact Registry, and Cloud Source Repositories
• Manage infrastructure-as-code using Terraform and Deployment Manager
• Maintain version control, automated testing and pipeline reliability with alerting and SLAs