Senior Data Engineer

iO Associates; City of Westminster; Hybrid; GBP 101,000 - 166,000; Full time

Job summary

iO Associates seeks an experienced Senior Data Engineer for a contract role in London with flexible remote work. You will own Azure Databricks-based data products within a product team, shaping analytics and AI capabilities.

You will design scalable data products, build and maintain production-grade pipelines on Databricks & Spark, and mentor engineers while raising engineering standards across the platform.

Qualifications

• 5+ years in data engineering or software engineering with a strong data focus.
• Experience building production-grade pipelines on Azure.
• Strong SQL and experience with analytical data models / data warehousing.
• Proficiency in Python (Scala a bonus).
• Experience delivering enterprise-grade data products used by multiple teams.
• Ability to communicate clearly with both technical and non-technical audiences.
• Experience with Azure Synapse (dedicated/serverless SQL pools).
• Agile mindset; experience working in Scrum teams.

Responsibilities

• Design scalable data products within a cross-functional product team.
• Act as a senior technical partner to product managers and lead data engineers.
• Build and maintain high-quality data pipelines using Databricks & Spark.
• Convert requirements into clear technical designs and conceptual data models.
• Ingest large, complex datasets and automate manual processes.
• Improve data quality, observability, reliability, and delivery performance.
• Champion a build-once-consume-many culture across the platform.
• Collaborate with stakeholders to solve data-related engineering challenges.
• Contribute to engineering standards, best practices, and platform evolution.
• Mentor other engineers and raise the overall engineering bar.

Skills

SQL

Python

Communication

Agile

Tools

Azure Databricks

Spark

Docker

Kubernetes

Delta Lake

Azure Synapse

Job description

My client is looking for an experienced Senior Data Engineer to join them on a contract basis, taking end‑to‑end ownership of business‑critical data products on their Azure Databricks‑based data platform. This is a hands‑on role embedded directly within a product team, working closely with product managers and lead engineers to raise engineering quality and shape how modern data products are built for analytics and AI.

The Role

As a Senior Data Engineer, you will:

Design scalable data products within a cross‑functional product team

Act as a senior technical partner to product managers and lead data engineers

Build and maintain high‑quality data pipelines using Databricks & Spark

Convert requirements into clear technical designs and conceptual data models

Ingest large, complex datasets and automate manual processes

Improve data quality, observability, reliability, and delivery performance

Champion a build‑once‑consume‑many culture across the platform

Collaborate with stakeholders to solve data‑related engineering challenges

Contribute to engineering standards, best practices, and platform evolution

Mentor other engineers and raise the overall engineering bar

Tech Environment

My client is actively modernising their platform, moving away from legacy IaaS solutions and building a Databricks‑first Lakehouse. Spark, Delta Lake, CI/CD, and modern DevOps practices are central to how they design and scale data products.

What You’ll Need

Strong hands‑on experience with Azure Databricks & Spark in production

5+ years in data engineering or software engineering with a strong data focus

Experience building production‑grade pipelines on Azure

Strong SQL and experience with analytical data models / data warehousing

Proficiency in Python (Scala a bonus)

Experience delivering enterprise‑grade data products used by multiple teams

Ability to communicate clearly with both technical and non‑technical audiences

Experience with Azure Synapse (dedicated/serverless SQL pools)

Agile mindset; experience working in Scrum teams

Nice to Have

Hands‑on experience with Docker, Kubernetes, or container orchestration

Start: ASAP

Location: London (with remote flexibility)

Length: 6 months (likely extension)

Rate: Competitive, depending on experience