Cloud Data Platform Lead — ETL/ELT & Airflow Expert

Vallum Associates; Sheffield; On-site; GBP 90,000 - 120,000; Full time

Job summary

Vallum Associates is seeking a Technical Lead to architect and deliver data engineering platforms in cloud environments. You will lead end-to-end data integration, design scalable ETL/ELT pipelines and champion CI/CD and Infrastructure-as-Code practices.

Strong Python skills, Airflow orchestration, and deep GCP experience with BigQuery, Pub/Sub, Dataflow, and container platforms are essential. You will help align teams and translate complex topics for diverse audiences.

Qualifications

• Proven experience as a Technical Lead/Lead Engineer delivering data platforms in cloud environments.
• Hands-on experience building production-grade ETL/ELT pipelines with Airflow.
• Strong Google Cloud Platform experience with BigQuery, Cloud Storage, Pub/Sub, Dataflow and/or container platforms.
• Experience with IAM, networking, secrets management and environment separation.
• Strong Python, APIs, configuration management, testing strategy and performance tuning.
• CI/CD expertise and Terraform for Infrastructure-as-Code.
• Designing for reliability with retries, idempotency, dead-letter patterns and backpressure.

Skills

Technical Lead

ETL/ELT pipelines

Python

APIs

Testing strategy

Performance tuning

CI/CD

Reliability design

Stakeholder alignment

Tools

Apache Airflow

Google Cloud Platform

BigQuery

Cloud Storage

Pub/Sub

Dataflow

GKE

Cloud Run

IAM

Networking

Secrets management

Terraform

Job description

Vallum Associates is seeking a Technical Lead to architect and deliver data engineering platforms in cloud environments. You will lead end-to-end data integration, design scalable ETL/ELT pipelines and champion CI/CD and Infrastructure-as-Code practices.

Strong Python skills, Airflow orchestration, and deep GCP experience with BigQuery, Pub/Sub, Dataflow, and container platforms are essential. You will help align teams and translate complex topics for diverse audiences.