Senior Data Engineer - Databricks Platform Builder

Oliver James Associates Ltd.; Greater London; On-site; GBP 86,000 - 117,000; Full time

Job summary

Oliver James Associates Ltd. is seeking a Senior Data Engineer to design, build and operate end-to-end data pipelines using Databricks in a hybrid London setting.

The role focuses on modernising a data platform within a transport/travel analytics transformation and delivering high-quality data products. You'll work in a self-organising team with strong emphasis on platform engineering, data modelling and APIs, while ensuring security and data protection throughout the project lifecycle.

Qualifications

• 5+ years’ progressive data engineering experience.
• Strong Databricks and Spark expertise – platform internals, performance, DAB, Delta Lake, declarative pipelines
• Advanced Python, with solid software/DevOps practice (version control, unit testing, pipelines-as-code, CI/CD)
• Excellent data modelling skills, comfortable with tradeoffs across technical and organisational lines (Conway's Law)
• Advanced SQL, with experience querying data lakes/lakehouses
• Experience ingesting data via APIs, database connections or other methods
• Strong Agile delivery experience in a self-organising team
• Clear, persuasive communication with both technical and non-technical stakeholders

Responsibilities

• Design, build and operate high-quality data pipelines end-to-end
• Lead on data modelling, working closely with business analysts and engineers to structure data for clarity, integrity and efficiency
• Be involved across the full project lifecycle – planning through to delivery and run
• Maintain a strong security and data-protection focus throughout

Skills

Databricks

Python

SQL

Agile delivery

Data modeling

Communication

Tools

Delta Lake

PostgreSQL

dbt

CI/CD

Job description

Oliver James Associates Ltd. is seeking a Senior Data Engineer to design, build and operate end-to-end data pipelines using Databricks in a hybrid London setting.

The role focuses on modernising a data platform within a transport/travel analytics transformation and delivering high-quality data products. You'll work in a self-organising team with strong emphasis on platform engineering, data modelling and APIs, while ensuring security and data protection throughout the project lifecycle.