Lead Data Engineer - Databricks for Real-Time Environmental Data

United Utilities; Warrington; Hybrid; GBP 90,000 - 135,000; Full time

Benefits offered by this job

Annual bonus

Pension

Private healthcare

Share save scheme

Wellbeing benefits

Job summary

United Utilities seeks a Lead Data Engineer to own Databricks-based data platforms for near real-time river water quality monitoring in Warrington. You will design, build and optimise scalable data pipelines across engineering and environmental teams.

You will lead technically, mentor engineers and ensure data governance, security and cost efficiency while embracing CI/CD and modern data architectures.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering or a related discipline.
• 5+ years' experience in data engineering roles, including 3+ years with Databricks.
• Proven ability to lead technically and mentor engineers.

Responsibilities

• Design scalable batch and streaming data pipelines using Databricks, Spark and Delta Lake.
• Implement medallion architecture (bronze/silver/gold) for data quality and lineage.
• Optimise Spark jobs to meet near real-time monitoring SLAs.
• Build and maintain streaming pipelines using Structured Streaming and Delta Live Tables.
• Ensure strong data governance, including schema enforcement and ACID compliance.
• Orchestrate pipelines using Databricks Jobs, APIs or cloud-native schedulers.
• Collaborate with data scientists, analysts and environmental specialists to deliver high-quality datasets.
• Monitor, troubleshoot and improve production pipelines for reliability and cost efficiency.
• Champion CI/CD, testing, version control and documentation.

Skills

Databricks (Spark/PySpark)

Delta Lake

Unity Catalog

Workflows

ETL/ELT pipelines

Python

SQL

Cloud platforms (Azure/AWS/GCP)

Data modelling

Data governance/security/compliance

Education

Bachelor's degree in Computer Science/Data Engineering

Tools

Airflow

CI/CD pipelines

Git workflows

Job description

United Utilities seeks a Lead Data Engineer to own Databricks-based data platforms for near real-time river water quality monitoring in Warrington. You will design, build and optimise scalable data pipelines across engineering and environmental teams.

You will lead technically, mentor engineers and ensure data governance, security and cost efficiency while embracing CI/CD and modern data architectures.