GCP Data Lead

HCL Technologies Limited; Adur; On-site; GBP 65,000 - 100,000; Full time

Job summary

HCL Technologies Limited is expanding its IDEA Lab in the UK to build, optimise, and scale cloud-native data pipelines. The Data Engineer will work on large-scale data processing workloads, supporting ingestion, transformation and analytics for the Business Transaction Banking platform.

The role requires experience with distributed data systems (Spark/Flink/Storm), data ingestion and streaming (Kafka/Pub/Sub), and strong programming in Java, Python or Scala.

Qualifications

• Bachelor's degree or higher in a related field.
• Hands-on experience with cloud data services (GCP preferred; Azure/AWS acceptable).
• Experience with distributed data processing frameworks (Spark, Flink, Storm).
• Strong knowledge of data ingestion, transformation and messaging systems (Kafka, Pub/Sub).
• Understanding of scalable data models and performance tuning for big data workloads.
• Knowledge of security for big-data systems including IAM and encryption.

Responsibilities

• Design and deliver end-to-end data pipelines on cloud platforms (GCP preferred).
• Build scalable ingestion, transformation, and processing workflows using distributed technologies.
• Develop ELT/ETL pipelines including legacy system decommissioning and modernisation.
• Work with relational, NoSQL, MPP and columnar databases (BigQuery, Redshift, SQLDW, etc.).
• Implement streaming pipelines using Kafka, Pulsar or Pub/Sub.
• Collaborate to optimise data models, partitioning, sharding and aggregation.

Skills

Java

Python

Scala

Cloud platforms

Distributed data processing

Kafka/Pub/Sub

Education

Bachelor's degree

Tools

Spark

Flink

Storm

BigQuery

Kubernetes

Airflow

Job description

Location: UK (Edinburgh)
Lab: IDEA – Innovation, Data Engineering & Artificial Intelligence

Summary

The IDEA Lab is expanding its GCP Data Products capability and is looking for a highly skilled Data Engineer to build, optimise, and scale data pipelines and large‑scale data processing workloads in a cloud‑native environment. You will work with modern distributed data systems, contribute to data platform modernisation, and support large‑scale ingestion, transformation, and analytics workloads across the Business Transaction Banking platform.

Key Responsibilities

Key Responsibilities

• Design and deliver end‑to‑end data pipelines on cloud platforms (GCP preferred).
• Build scalable data ingestion, transformation, and processing workflows using distributed technologies such as Spark, Flink, Storm, or similar.
• Develop robust ELT/ETL processing and migration pipelines, including support for legacy Datastage decommissioning and modernisation.
• Work with a variety of database technologies including relational, NoSQL, MPP and columnar stores (BigQuery, Redshift, Azure SQLDW, HBase, MongoDB).
• Implement streaming and messaging‑based pipelines using Kafka, Pulsar or Pub/Sub.
• Build optimised, scalable data models to support diverse consumption patterns, applying partitioning, sharding, bucketing and aggregation strategies.
• Apply performance tuning and optimisation across storage, compute and query layers.
• Ensure secure handling of data including authentication, authorisation, encryption in transit/at rest, and cloud‑native security controls.
• Implement monitoring, alerting and observability for large‑scale distributed data workloads.
• Use orchestration tools such as Cloud Composer, Airflow or equivalent to operationalise pipelines.
• Contribute to CI/CD, containerisation, Kubernetes‑based deployments, and automated testing practices.
• Participate in data governance, metadata, catalogue and lineage processes as needed.
• Collaborate with engineers, architects and SMEs to deliver stable, high‑quality data products.

Skill Requirements

Required Skills & Experience

• Strong programming skills in Java (preferred), Python, or Scala.
• Hands‑on experience with cloud data services (GCP preferred; Azure/AWS acceptable).
• Practical experience with distributed data processing frameworks such as Spark (Core/SQL/Streaming), Flink, or Storm.
• Strong knowledge of data ingestion, transformation and messaging systems: Kafka, Pulsar, Pub/Sub, etc.
• Understanding of designing scalable data models for varied access patterns.
• Experience with performance tuning, cost‑optimisation and scaling strategies.
• Experience delivering large‑scale big data solutions in batch and/or streaming environments, on cloud or on‑premise.
• Good familiarity with the wider data ecosystem and open‑source frameworks.
• Experience with orchestration (Airflow/Composer) and workflow automation.
• Understanding of DevOps for data systems: CI/CD, containers, Kubernetes and automated testing.
• Knowledge of security for big‑data systems including IAM, encryption, and cluster‑level controls.
• Basic understanding of monitoring and alerting for distributed systems.
• Knowledge of dimensional modelling (star, snowflake, normalized/denormalized).
• Awareness of data governance, cataloguing and lineage tools.

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 223,000 people across 60 countries, delivering industry‑leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending June 2026totaled $14.8billion.