Data Integration Platform Lead

Vallum Associates; Sheffield; On-site; GBP 90,000 - 130,000; Full time

Job summary

Vallum Associates seeks a Technical Lead / Lead Engineer to drive the design and delivery of data engineering platforms in cloud environments. You will hands-on build production-grade ETL/ELT pipelines with Apache Airflow and shape scalable cloud architectures using Google Cloud Platform services like BigQuery, Cloud Storage and Dataflow.

Leadership duties include mentoring engineers, aligning cross-team efforts, and advancing CI/CD and IaC practices (Terraform) to ensure reliability, security,

Qualifications

• Proven experience as a Technical Lead / Lead Engineer delivering data engineering or integration platforms in cloud environments.
• Strong hands-on experience building production-grade ETL/ELT pipelines and orchestration with Apache Airflow.
• Solid experience on Google Cloud Platform with Data services such as BigQuery, Cloud Storage, Pub/Sub, Dataflow and/or container platforms like GKE, Cloud Run.
• IAM, networking, secrets management, and environment separation.
• Strong software engineering foundations: Python, APIs, configuration management, testing strategy, and performance tuning.
• CI/CD expertise and Infrastructure-as-Code (Terraform).
• Experience designing for reliability: retries, idempotency, dead-letter patterns, backpressure handling, and operational resilience.
• Ability to translate complex technical topics for mixed audiences and drive alignment across teams.

Responsibilities

• Lead the design and delivery of data engineering or integration platforms in cloud environments.
• Oversee production-grade ETL/ELT pipelines and Airflow orchestration.
• Ensure secure, scalable cloud architectures with IAM, networking, and secret management.
• Drive CI/CD practices and IaC using Terraform; establish reliability patterns.
• Mentor engineers and align cross-team initiatives on data platforms.

Skills

Lead engineer

Data engineering

ETL/ELT pipelines

Airflow

Google Cloud Platform

Cloud data services

BigQuery

Cloud Storage

Pub/Sub

Dataflow

GKE

Cloud Run

IAM

Networking

Secrets management

Environment separation

Python

APIs

Configuration management

Testing strategy

Performance tuning

CI/CD

Terraform

Infrastructure-as-Code

Reliability design

Backoff/retries

Dead-letter patterns

Tools

Apache Airflow

Terraform

Google Cloud Platform

BigQuery

Cloud Storage

Pub/Sub

Dataflow

GKE

Cloud Run

Job description

• Proven experience as a Technical Lead / Lead Engineer delivering data engineering or integration platforms in cloud environments.
• Strong hands-on experience building production-grade ETL/ELT pipelines and orchestration with Apache Airflow.
• Solid experience on Google Cloud Platform, such as:
• Data services (e.g., BigQuery, Cloud Storage, Pub/Sub, Dataflow) and/or container platforms (e.g., GKE, Cloud Run)
• IAM, networking, secrets management, and environment separation
• Strong software engineering foundations: Python, APIs, configuration management, testing strategy, and performance tuning.
• CI/CD expertise (pipeline tooling, release controls, automated quality gates) and Infrastructure-as-Code experience (e.g., Terraform).
• Experience designing for reliability: retries, idempotency, dead-letter patterns, backpressure handling, and operational resilience.
• Ability to translate complex technical topics for mixed audiences and drive alignment across teams.