Senior Data Engineer (Distributed Data Processing)

Xcede; Grimsby; Remote; GBP 90,000 - 120,000; Full time

Job summary

A leading data technology firm is seeking a Senior Data Engineer to design and build large-scale distributed data pipelines. This fully remote role requires strong Python and Spark skills, along with experience in cloud infrastructure. You will work independently while mentoring other engineers and optimizing data workflows. Ideal candidates will have real-world experience with Databricks and a proven track record in handling large-scale data processing in regulated industries.

Qualifications

• Proven experience as a Senior Data Engineer.
• Strong Python software engineering foundation.
• Hands-on Spark experience in production (PySpark essential).
• Real-world experience using Databricks for data pipelines.

Responsibilities

• Design, build, and evolve large-scale distributed data pipelines using Spark/PySpark.
• Develop production-grade Python data workflows.
• Optimize cloud-based data infrastructure (AWS preferred).
• Collaborate with engineers on client-facing projects.
• Support and mentor other engineers.
• Collaborate with engineers and delivery teams on client projects.
• Lead by example and mentor other engineers.

Skills

Python

Spark

Databricks

Cloud Infrastructure (AWS/Azure)

Large-scale data processing

Distributed data processing

Mentoring

Tools

Databricks

AWS

Job description

Senior Data Engineer (Distributed Data Processing)

UK (O/IR35), Belgium, Netherlands or Germany (B2B) • Fully Remote • Contract • Great Work, England, United Kingdom

We’re looking for a Senior Data Engineer to join a data-intensive SaaS platform operating in a complex, regulated industry. This is a hands‑on senior IC role focused on distributed data processing, Spark‑based pipelines, and Python‑heavy engineering. You’ll be working on large‑scale batch data workflows that power pricing, forecasting, and operational decision‑making systems. The role requires strong engineering judgement, the ability to operate autonomously, and the confidence to mentor others while delivering under tight timelines. This is not an ML, Data Science, or GenAI role.

What You’ll Be Doing

• Design, build, and evolve large‑scale distributed data pipelines using Spark / PySpark.
• Develop production‑grade Python data workflows that implement complex business logic.
• Work with Databricks for job execution, orchestration, and optimisation.
• Own and optimise cloud‑based data infrastructure (AWS preferred, Azure also relevant).
• Optimise data workloads for performance, reliability, and cost.
• Collaborate with engineers, domain specialists, and delivery teams on client‑facing projects.
• Take ownership of technical initiatives and lead by example within the team.
• Support and mentor other engineers.

Must‑Have Experience

• Proven experience as a Senior Data Engineer.
• Strong Python software engineering foundation.
• Hands‑on Spark experience in production (PySpark essential).
• Real‑world experience using Databricks for data pipelines (Spark depth matters most).
• Experience with large‑scale or parallel data processing.
• Ownership of cloud infrastructure (AWS and/or Azure).
• Comfortable operating with senior‑level autonomy and responsibility.
• Experience mentoring or supporting other engineers.

Nice‑to‑Have Experience

• Experience working with time‑series data.
• Background in utilities, energy, or other data‑heavy regulated industries.
• Exposure to streaming technologies (Kafka, event‑driven systems), though the role is primarily batch‑focused.