Lead Data Engineer – Real-time Databricks Pipelines

United Utilities; Warrington; Hybrid; GBP 90,000 - 110,000; Full time

Benefits offered by this job

Up to 10.5% annual bonus

Outstanding pension – up to 21%

Wellbeing benefits and support

Job summary

United Utilities is seeking a Lead Data Engineer to own Databricks-based data platforms for near real-time river water quality insights. You will design and optimise scalable batch and streaming pipelines, ensure strong data governance, and collaborate with scientists and analysts to deliver trusted datasets.

In Warrington, this hybrid role combines on-site collaboration with remote work flexibility, offering leadership responsibilities and a clear path to uplift data capabilities across the

Qualifications

• 5+ years’ experience in data engineering roles, including 3+ years with Databricks.
• Bachelor’s degree in Computer Science, Data Engineering or related discipline.

Responsibilities

• Design, build and optimise Databricks-based data pipelines for batch and streaming data.
• Optimise Spark jobs to meet near real-time monitoring SLAs.
• Build and maintain streaming pipelines with Structured Streaming and Delta Live Tables.
• Ensure data governance, schema enforcement, validation rules and ACID compliance.
• Orchestrate pipelines with Databricks Jobs, APIs or cloud schedulers.
• Collaborate with data scientists and analysts to deliver high-quality datasets.
• Monitor, troubleshoot and continuously improve production pipelines for reliability and cost efficiency.
• Champion CI/CD, testing, version control and documentation best practices.
• Act as technical leader and mentor to engineers.

Skills

Databricks

Spark

PySpark

Delta Lake

Unity Catalog

Workflows

ETL/ELT pipelines

Python

SQL

Cloud platforms

Data modelling

Data governance

Airflow

CI/CD

Git workflows

Automated testing

IoT data

Education

Bachelor’s degree in Computer Science or Data Engineering

Tools

Airflow

Job description

United Utilities is seeking a Lead Data Engineer to own Databricks-based data platforms for near real-time river water quality insights. You will design and optimise scalable batch and streaming pipelines, ensure strong data governance, and collaborate with scientists and analysts to deliver trusted datasets.

In Warrington, this hybrid role combines on-site collaboration with remote work flexibility, offering leadership responsibilities and a clear path to uplift data capabilities across the