Data Engineer

Severn Trent; Deutschland; Hybrid; EUR 76.000 - 111.000; Vollzeit

Zusammenfassung

Severn Trent is seeking a data engineering professional to design, build and operate PySpark-based pipelines on Databricks, and to implement governed ingestion flows with Informatica IDMC. You will model bronze/silver/gold datasets, ensure data quality and optimise performance and cost across cloud environments.

The role supports migration from Azure Synapse during a transition period, contributing to both net-new delivery and stabilization of existing workloads.

Qualifikationen

• Strong experience building PySpark-based pipelines that run reliably in production.
• Solid grasp of Spark execution behaviour including partitions, shuffles, joins and performance trade-offs.
• Hands-on experience with Databricks in a real delivery environment.
• Experience using Informatica IDMC or equivalent enterprise integration tooling to build governed ingestion flows.
• Strong SQL skills and confidence working across raw, curated and consumer-ready data layers.

Aufgaben

• Databricks pipeline engineering: Design, build and operate Spark pipelines on Databricks, including handling schema evolution, incremental loads, backfills, and failure recovery.
• Integration and orchestration (IDMC): Implement governed ingestion patterns from source systems into curated lakehouse layers.
• Lakehouse modelling: Build bronze, silver and gold datasets for downstream analytics and data products.
• Data quality and reliability: Implement automated data quality checks, validations and controls to prevent bad data reaching consumers.
• Performance and cost optimisation: Tune Spark workloads, storage layouts and execution patterns with cloud cost awareness.
• Dual-run and migration delivery: Support Synapse workloads while contributing to Databricks migrations and new delivery.
• Technical leadership: Set engineering standards through code reviews and practical mentorship.

Kenntnisse

PySpark pipelines

Spark execution behaviour

Databricks

Informatica IDMC

SQL

Data layers (raw/ curated/ consumer)

Tools

Databricks

Informatica IDMC

Jobbeschreibung

• Databricks pipeline engineering: Design, build and operate Spark pipelines on Databricks, including handling schema evolution, incremental loads, backfills, and failure recovery.
• Integration and orchestration (IDMC): Use Informatica IDMC to implement governed, repeatable ingestion patterns from source systems into curated lakehouse layers.
• Lakehouse modelling: Build curated bronze, silver and gold datasets that are performant, testable, and fit for downstream analytics and data products.
• Data quality and reliability: Implement automated data quality checks, validations and controls that prevent bad data from silently reaching consumers.
• Performance and cost optimisation: Actively tune Spark workloads, storage layouts and execution patterns with an awareness of cloud cost and runtime behaviour.
• Dual-run and migration delivery: Support and stabilise existing Synapse workloads while contributing to migration and net-new delivery on Databricks.
• Technical leadership: Set engineering standards through code reviews, patterns and practical mentorship rather than documentation alone.

You'll be based at our Severn Trent Centre head office in Coventry. You'll work within our dedicated Data Engineering team of 14. With this being such a critical role, we're looking for someone who can join us 37 hours a week, Monday to Friday.

HOW WE WORK

You'll join a caring culture that collaborates to achieve, grow, and develop. Our employee engagement scores are among the highest globally in energy and utilities. That's why, we value in-person moments to keep our culture alive but also understand the flexibility working from home can bring. So, you'll usually find us in the office, but working from home is supported, when you need it.

Experience operating or supporting Azure Synapse is useful during the transition, but it is not the long-term focus of the role.

We are looking for engineers who are genuinely strong at data engineering as a craft, not generalists who have only used platforms at a surface level.

You will have:

• Strong experience building PySpark-based pipelines that run reliably in production.
• A solid grasp of Spark execution behaviour, including partitioning, shuffles, joins and performance trade-offs.
• Hands-on experience with Databricks in a real delivery environment.
• Experience using Informatica IDMC or equivalent enterprise integration tooling to build governed ingestion flows.
• Strong SQL skills and confidence working across raw, curated and consumer-ready data layers.

Skills and experience are important, but character, positivity, and a caring attitude matter too. We welcome people from all walks of life and celebrate individuality as we know diverse minds, experiences and backgrounds help us to learn and better serve our communities. We seek people who get involved, want to be part of something bigger, and make a difference because they care.

At Severn Trent, our people are at the heart of everything we do. We're in the top 5% of utility companies