GCP Data Platform Architect: ETL/ELT & Real-Time Pipelines

HCL Technologies Limited; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

HCL Technologies Limited is seeking a skilled data engineer to design, develop and operate scalable ETL/ELT pipelines across structured and unstructured data in a GMP-like cloud environment. You will build batch and real-time processing solutions using BigQuery, Dataflow, Cloud Storage, Pub/Sub and related GCP services, while delivering robust data models and cost-aware processing patterns.

You will implement reusable data ingestion and transformation frameworks, ensure data quality and

Qualifications

• Degree in Computer Science, Software Engineering or a related discipline, or equivalent practical experience.
• Experience delivering software solutions in agile teams.
• Knowledge of modern development frameworks, tools and engineering practices.

Responsibilities

• Design, develop, test, deploy, and maintain scalable ETL/ELT pipelines for structured and unstructured data.
• Build batch and real-time processing solutions using BigQuery, Dataflow, Cloud Storage, Pub/Sub, and related GCP services.
• Develop data-processing components and integrations using Java or Python, SQL, REST APIs, Kafka, and containerized services on GKE.
• Implement reusable ingestion and transformation frameworks supporting full, incremental, and event-driven loads.
• Design data models and optimize BigQuery tables through partitioning, clustering, efficient SQL, and cost-aware processing patterns.
• Produce high-level and low-level technical designs, document assumptions, and contribute to architecture reviews.
• Implement data-quality checks, schema validation, reconciliation, audit logging, lineage, monitoring, alerting, and error-handling mechanisms.
• Troubleshoot complex production issues, perform root-cause analysis, and deliver sustainable corrective actions.
• Apply security, privacy, access-control, reliability, and operational-support standards across data solutions.
• Use source control, automated testing, CI/CD, and infrastructure-as-code practices to promote repeatable deployments.
• Participate in code reviews, improve engineering standards, and mentor less-experienced engineers.
• Collaborate with architects, analysts, application teams, platform teams, and business stakeholders to deliver reliable data products.

Skills

BigQuery

Dataflow

Pub/Sub

Cloud Storage

SQL

Java

Python

REST APIs

Kafka

Docker

Kubernetes

GKE

JSON

Parquet

Git

CI/CD

Observability

Release management

Education

Computer Science degree

Software Engineering degree

Equivalent practical experience

Tools

Cloud Composer

Apache Airflow

Dataproc

Spark

Dataform

dbt

Terraform

Job description

HCL Technologies Limited is seeking a skilled data engineer to design, develop and operate scalable ETL/ELT pipelines across structured and unstructured data in a GMP-like cloud environment. You will build batch and real-time processing solutions using BigQuery, Dataflow, Cloud Storage, Pub/Sub and related GCP services, while delivering robust data models and cost-aware processing patterns.

You will implement reusable data ingestion and transformation frameworks, ensure data quality and