Data Engineering Lead — Hybrid, Databricks & AWS Scale

The Vanguard Group; Manchester; Hybrid; GBP 90,000 - 120,000; Full time

Job summary

Vanguard is seeking an Data Engineering Technical Lead to shape and deliver Europe-wide data and analytics capabilities. You will lead the technical design and evolution of data products supporting multiple business units within the CDAO.

You will guide the Databricks-based data lake, batch and streaming pipelines, and CI/CD practices, mentoring engineers and ensuring high-quality, scalable solutions.

Qualifications

• Must-have: Databricks, PySpark, SQL, Apache Spark expertise.
• Cloud data platform and lakehouse design experience.
• Hands-on Delta Lake, Unity Catalog, Databricks Workflows; performance tuning.

Responsibilities

• Design and build scalable data pipelines using Databricks, PySpark, SQL, AWS Glue and Lambda.
• Develop and manage Data Lake solutions with Bronze/Silver/Gold architectures.
• Implement batch and real-time data processing with Spark Structured Streaming and Auto Loader.

Skills

Databricks

PySpark

SQL

Apache Spark

Cloud data platforms

Delta Lake/Unity Catalog

Data pipelines (batch & streaming)

Data modelling & product design

Terraform

Git

CI/CD

Automated testing

Leadership & stakeholder management

Job description

Vanguard is seeking an Data Engineering Technical Lead to shape and deliver Europe-wide data and analytics capabilities. You will lead the technical design and evolution of data products supporting multiple business units within the CDAO.

You will guide the Databricks-based data lake, batch and streaming pipelines, and CI/CD practices, mentoring engineers and ensuring high-quality, scalable solutions.