Gcp Data Engineer

Lloyds Technology Centre; Hyderabad; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Lloyds Technology Centre in Hyderabad is seeking a Senior Data Engineer specialized in GCP, BigQuery, PySpark, and Kafka to design, build, and optimize scalable data platforms. You will work on end-to-end data pipelines, including real-time streams and batch processing, with Terraform and Kubernetes for infrastructure and deployments.

The role requires strong hands-on skills in Python, SQL, ETL/ELT, and DBT, with a focus on data governance, quality, and reliability across enterprise-scale

Qualifications

• Bachelor's or Master’s degree in CS/IT/Engineering or related field.
• 5-13 years of experience in Data Engineering and Cloud-based Data Platforms.
• Strong hands-on expertise in GCP ecosystem and BigQuery.
• Experience building enterprise-grade data platforms and large-scale distributed data processing systems.
• Strong understanding of cloud-native architectures and modern data engineering practices.
• Excellent problem-solving and analytical skills.
• Strong communication and stakeholder management capabilities.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing frameworks on GCP.
• Build and optimize batch and real-time ETL/ELT pipelines using Python, PySpark, Kafka, and BigQuery.
• Develop high-performance data solutions for large-scale structured and unstructured datasets.
• Create and maintain data models using DBT and ensure data governance standards.
• Design streaming solutions using Kafka and event-driven architectures.
• Implement IaC using Terraform for automated cloud provisioning.
• Deploy and manage containerized applications using Kubernetes.
• Build automated CI/CD pipelines for data platform deployments and releases.
• Optimize BigQuery performance, partitioning, clustering, and query tuning.
• Collaborate with data scientists, analysts, architects, and business stakeholders to build data products.
• Troubleshoot production issues and drive continuous improvements in platform reliability and performance.
• Ensure data quality, security, and operational excellence across the data ecosystem.

Skills

Python

PySpark

Advanced SQL

ETL / ELT Pipeline Development

Data Warehousing Concepts

Data Modeling

BigQuery

Dataflow

Pub/Sub

Cloud Composer

Cloud Storage

Cloud Functions

Apache Kafka

Event-Driven Architecture

Real-Time Data Processing

Terraform

Kubernetes

Docker

CI/CD Pipelines

Git

DBT (Data Build Tool)

Education

Bachelor's or Master's degree in Computer Science, IT, Engineering

Tools

Kubernetes

Docker

Terraform

Git/GitHub/GitLab

Airflow/Cloud Composer

Job description

Senior Data Engineer - GCP, Big Query, PySpark & Kafka

Location

Hyderabad (Hybrid)

Experience

5 to 13 Years

Role Overview

We are looking for a highly skilled Data Engineer with strong expertise in building scalable, cloud-native data platforms on Google Cloud Platform (GCP). The ideal candidate should have extensive hands-on experience with Python, PySpark, BigQuery, Kafka, DBT, Terraform, Kubernetes, SQL, CI/CD, and ETL pipeline development. The candidate will be responsible for designing, developing, optimizing, and supporting large-scale data processing solutions that drive business insights and analytics.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing frameworks on GCP.
• Build and optimize batch and real-time ETL/ELT pipelines using Python, PySpark, Kafka, and BigQuery.
• Develop high-performance data solutions for large-scale structured and unstructured datasets.
• Create and maintain data models using DBT and ensure adherence to data governance standards.
• Design streaming solutions using Kafka and event-driven architectures.
• Implement infrastructure as code (IaC) using Terraform for automated cloud provisioning.
• Deploy and manage containerized applications using Kubernetes.
• Build automated CI/CD pipelines for data platform deployments and releases.
• Optimize BigQuery performance, partitioning, clustering, and query tuning.
• Collaborate with data scientists, analysts, architects, and business stakeholders to build data products.
• Troubleshoot production issues and drive continuous improvements in platform reliability and performance.
• Ensure data quality, scalability, security, and operational excellence across the data ecosystem.

Must Have Skills

Data Engineering

• Python
• PySpark / Apache Spark
• Advanced SQL
• ETL / ELT Pipeline Development
• Data Warehousing Concepts
• Data Modeling

Google Cloud Platform (GCP)

• BigQuery
• Dataflow
• Pub/Sub
• Cloud Composer
• Cloud Storage (GCS)
• Cloud Functions (Good to Have)

Streaming & Messaging

• Apache Kafka
• Event-Driven Architecture
• Real-Time Data Processing

DevOps & Cloud Engineering

• Terraform
• Kubernetes
• Docker
• CI/CD Pipelines
• Git/GitHub/GitLab

Data Transformation

• DBT (Data Build Tool)

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.
• 5-13 years of experience in Data Engineering and Cloud-based Data Platforms.
• Strong hands-on expertise in GCP ecosystem and BigQuery.
• Experience building enterprise-grade data platforms and large-scale distributed data processing systems.
• Strong understanding of cloud-native architectures and modern data engineering practices.
• Excellent problem-solving and analytical skills.
• Strong communication and stakeholder management capabilities.

Preferred Skills

• Experience with Data Lake and Lakehouse architectures.
• Knowledge of Airflow/Cloud Composer.
• Exposure to BI/Analytics platforms such as Tableau, Looker, or Power BI.
• Experience in Financial Services, Banking, Insurance, or Enterprise Data Platforms.
• Exposure to Data Governance and Data Quality tools.

What We Are Looking For

• Strong hands-on coding experience in Python and PySpark.
• Deep expertise in BigQuery performance optimization.
• Proven experience with Kafka-based streaming solutions.
• Strong Terraform and Kubernetes implementation experience.
• Experience building robust CI/CD pipelines for data platforms.
• Ability to design end-to-end cloud-native data solutions on GCP.