GCP Data Engineer

Infosys; Dadri; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Infosys is hiring a seasoned GCP Data Engineer to design and implement scalable cloud data platforms. You will build batch and streaming pipelines using BigQuery, Dataflow, Dataproc, PySpark, and Databricks, focusing on performance and cost optimization.

You will optimize Delta Lake workloads, implement CI/CD with Cloud Build and Terraform, ensure data quality and governance, and collaborate across Agile teams to deliver analytics-ready datasets for global clients.

Qualifications

• Strong hands-on GCP Data Engineering experience across scalable cloud data platforms.
• Expertise in Google Cloud Storage, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, Cloud Functions, Cloud Monitoring, IAM and Secret Manager.
• Strong experience in batch and real-time streaming using Pub/Sub, Dataflow / Apache Beam, Spark Structured Streaming or Kafka.
• Hands-on coding proficiency in Python, PySpark and SQL, including debugging and performance optimization.
• Experience building high-volume, low-latency ingestion and transformation pipelines with retry, idempotency, deduplication and dead-letter handling.
• Strong experience in Databricks, Delta Lake, Structured Streaming, Auto Loader and Spark performance tuning.
• Experience handling TB/PB-scale datasets, partitioning, clustering, file compaction, schema evolution and data lifecycle management.
• Strong understanding of Data Lake, Lakehouse and modern cloud data-platform architectures.
• Practical data-modeling experience for analytics, reporting and downstream consumption.
• Experience implementing CI/CD using Cloud Build, GitHub Actions, Jenkins or equivalent, with Terraform for Infrastructure as Code.
• Strong troubleshooting, observability, production support and cloud cost-optimization skills.
• Excellent client communication, stakeholder management and problem-solving skills.

Responsibilities

• Design and develop scalable GCP Data Engineering solutions and cloud-native data platforms.
• Build batch and real-time pipelines using BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark and Databricks.
• Develop and optimize Databricks, Delta Lake, Spark and Structured Streaming workloads for large-scale processing.
• Implement high-throughput streaming solutions using Pub/Sub and Dataflow, including windowing, watermarking, late-arriving data, duplicate handling and dead-letter queues.
• Handle high-volume TB/PB-scale data with focus on performance, scalability, reliability and cost optimization.
• Develop complex SQL transformations, data models and analytics-ready BigQuery datasets using partitioning and clustering.
• Orchestrate pipelines using Cloud Composer / Airflow and implement CI/CD and Terraform-based Infrastructure as Code.
• Ensure data quality, governance, security, monitoring, alerting and production support.
• Collaborate with architects, stakeholders and cross-functional teams in an Agile delivery model.
• Troubleshoot and optimize end-to-end GCP, Databricks and streaming data pipelines.

Skills

GCP Data Engineering

BigQuery

Dataflow

Dataproc

Pub/Sub

Databricks

Delta Lake

Structured Streaming

PySpark

Python

SQL

Cloud Composer

Cloud Functions

IAM & Secret Manager

CI/CD

Terraform

Monitoring & Observability

Cost Optimization

Education

Master Of Comp. Applications

Master Of Technology

Bachelor Of Science

Bachelor Of Engineering

Bachelor Of Technology

Tools

Databricks

Delta Lake

Spark

Airflow / Cloud Composer

Terraform

Cloud Build

GitHub Actions

Job description

Job Description

• Strong hands-on GCP Data Engineering experience across scalable cloud data platforms.
• Expertise in Google Cloud Storage, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, Cloud Functions, Cloud Monitoring, IAM and Secret Manager.
• Strong experience in batch and real-time streaming using Pub/Sub, Dataflow / Apache Beam, Spark Structured Streaming or Kafka.
• Hands-on coding proficiency in Python, PySpark and SQL, including debugging and performance optimization.
• Experience building high-volume, low-latency ingestion and transformation pipelines with retry, idempotency, deduplication and dead-letter handling.
• Strong experience in Databricks, Delta Lake, Structured Streaming, Auto Loader and Spark performance tuning.
• Experience handling TB/PB-scale datasets, partitioning, clustering, file compaction, schema evolution and data lifecycle management.
• Strong understanding of Data Lake, Lakehouse and modern cloud data-platform architectures.
• Practical data-modeling experience for analytics, reporting and downstream consumption.
• Experience implementing CI/CD using Cloud Build, GitHub Actions, Jenkins or equivalent, with Terraform for Infrastructure as Code.
• Strong troubleshooting, observability, production support and cloud cost-optimization skills.
• Excellent client communication, stakeholder management and problem-solving skills.

Roles & Responsibilities

• Design and develop scalable GCP Data Engineering solutions and cloud-native data platforms.
• Build batch and real-time pipelines using BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark and Databricks.
• Develop and optimize Databricks, Delta Lake, Spark and Structured Streaming workloads for large-scale processing.
• Implement high-throughput streaming solutions using Pub/Sub and Dataflow, including windowing, watermarking, late-arriving data, duplicate handling and dead-letter queues.
• Handle high-volume TB/PB-scale data with focus on performance, scalability, reliability and cost optimization.
• Develop complex SQL transformations, data models and analytics-ready BigQuery datasets using partitioning and clustering.
• Orchestrate pipelines using Cloud Composer / Airflow and implement CI/CD and Terraform-based Infrastructure as Code.
• Ensure data quality, governance, security, monitoring, alerting and production support.
• Collaborate with architects, stakeholders and cross-functional teams in an Agile delivery model.
• Troubleshoot and optimize end-to-end GCP, Databricks and streaming data pipelines.

Technical Requirement

1. GCP BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, batch and real-time streaming, Databricks , Pyspark
2. Python, strong SQL
3. Databricks with Delta Lake and Structured Streaming

Additional Responsibility

Infosys is a global leader in next-generation digital services and consulting. Within the Data & Analytics unit, you will work on cutting-edge data engineering and analytics initiatives for global clients across industries. The role offers opportunities to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and large-scale enterprise data modernization programs. Employees benefit from continuous learning programs, certifications, and career growth opportunities within Infosys' data and analytics practice

Educational Requirement

• Master Of Comp. Applications
• Master Of Technology
• Bachelor Of Science
• Bachelor Of Engineering
• Bachelor Of Technology

Preferred Skills

Technology->Cloud Platform->GCP Database

Service Line

Data & Analytics Unit