Senior Data Engineer

Oliver James; Greater London; Hybrid; GBP 96,000 - 111,000; Full time

Job summary

Oliver James in London is seeking a Senior Data Engineer to design, build and operate end-to-end data pipelines on Databricks within a hybrid London setup. You will lead data modelling initiatives and collaborate with analysts and engineers to structure data for analytics.

The role requires strong Python, SQL, and APIs experience, plus CI/CD and pipelines-as-code practices in an agile team. 12-month contract with a daily rate of £550 outside IR35.

Qualifications

• 5+ years' progressive data engineering experience.
• Strong Databricks and Spark expertise; platform internals, Delta Lake.
• Advanced Python with pipelines-as-code and CI/CD.
• Excellent data modelling skills and stakeholder communication.
• Experience querying data lakes/lakehouses and APIs ingestion.

Responsibilities

• Design, build, and operate end-to-end data pipelines.
• Lead data modelling with business analysts and engineers.
• Oversee full project lifecycle from planning to delivery.
• Maintain security and data protection throughout.

Skills

Databricks

Spark

Python

SQL

APIs

Agile delivery

Data modelling

Stakeholder communication

Tools

Databricks

Spark

dbt

PostgreSQL

Job description

Senior Data Engineer - Databricks | Hybrid London | £550/day Outside IR35 | 12 Months

A well-established transport/travel sector business is expanding its Data & Middleware function as part of a major analytics transformation, and is looking for a Senior Data Engineer to help build out its data platform and data products.

You'll join a growing team modernising a legacy estate onto a modern Event Sourcing Architecture, producing high-quality data for analytics via a flexible, platform-engineering-led approach. This is a genuine build role - shaping the technical direction of the platform, not just maintaining what's there.

The role:

• Design, build and operate high-quality data pipelines end-to-end
• Lead on data modelling, working closely with business analysts and engineers to structure data for clarity, integrity and efficiency
• Be involved across the full project lifecycle - planning through to delivery and run
• Maintain a strong security and data-protection focus throughout

What you'll need:

• 5+ years' progressive data engineering experience
• Strong Databricks and Spark expertise - platform internals, performance, DAB, Delta Lake, declarative pipelines
• Advanced Python, with solid software/DevOps practice (version control, unit testing, pipelines-as-code, CI/CD)
• Excellent data modelling skills, comfortable with tradeoffs across technical and organisational lines (Conway's Law)
• Advanced SQL, with experience querying data lakes/lakehouses
• Experience ingesting data via APIs, database connections or other methods
• Strong Agile delivery experience in a self-organising team
• Clear, persuasive communication with both technical and non-technical stakeholders

Nice to have:

platform engineering exposure, event sourcing/data modelling experience, ML/AI operationalisation on data platforms, PostgreSQL, dbt.

Details:

Hybrid, 2-3 days/week in London. 12-month contract, £550/day outside IR35.