Lead Data Engineer

iO Associates; Greater London; On-site; GBP 120,000 - 170,000; Full time

Job summary

iO Associates is seeking an experienced Lead Data Engineer to drive design, delivery, and evolution of a modern data platform built on Databricks, Microsoft Fabric, Azure Synapse and Azure. This hands-on leadership role guides a team of Data Engineers, sets engineering standards, and delivers scalable, secure, high-performing data solutions.

You’ll lead a high-performing engineering team, act as the technical SME across Databricks, Fabric, Synapse and Azure, own solution design and delivery, and

Qualifications

• Proven leadership of Data Engineering teams.
• Strong commercial experience with Databricks, Fabric, or Azure Synapse.
• Deep expertise in PySpark, Spark SQL, and SQL optimisation.
• ETL/ELT pipeline development and orchestration.
• Experience building scalable cloud-based data platforms.

Responsibilities

• Lead and mentor a high-performing Data Engineering team.
• Act as the technical SME across Databricks, Fabric, Synapse, and Azure.
• Own solution design, implementation strategy, and technical delivery.
• Build and optimise pipelines using PySpark, Spark SQL, or Fabric Pipelines.
• Define engineering best practice, conduct code reviews, and uplift quality.
• Develop secure, performant SQL solutions.
• Manage Azure DevOps Boards, Repos, and CI/CD Pipelines.
• Collaborate with architects, product owners, and stakeholders.
• Drive Agile delivery, sprint planning, and continuous improvement.

Skills

Leadership

Databricks

Microsoft Fabric

Azure Synapse

PySpark

Spark SQL

SQL optimisation

ETL/ELT pipelines

Cloud-based data platforms

Azure DevOps / CI/CD

Tools

Azure DevOps

Azure DevOps Boards

CI/CD pipelines

Job description

My client is seeking an experienced Lead Data Engineer to drive the design, delivery, and evolution of a modern data platform built on Databricks, Microsoft Fabric, Azure Synapse, and Azure. This is a hands-on technical leadership role where you’ll guide a team of Data Engineers, set engineering standards, and deliver scalable, secure, high-performing data solutions.

What you’ll lead

Lead and mentor a high-performing Data Engineering team.

Act as the technical SME across Databricks, Fabric, Synapse, and Azure.

Own solution design, implementation strategy, and technical delivery.

Build and optimise pipelines using PySpark, Spark SQL, ADF, Synapse Pipelines, or Fabric Pipelines.

Define engineering best practice, conduct code reviews, and uplift quality.

Develop secure, performant SQL solutions.

Manage Azure DevOps Boards, Repos, and CI/CD Pipelines.

Collaborate with architects, product owners, and stakeholders.

Drive Agile delivery, sprint planning, and continuous improvement.

What you’ll bring

Proven leadership of Data Engineering teams.

Strong commercial experience with Databricks, Microsoft Fabric, or Azure Synapse.

Deep expertise in PySpark, Spark SQL, and SQL optimisation.

ETL/ELT pipeline development and orchestration.

Experience building scalable cloud-based data platforms.

Nice to Have

Snowflake experience.

Broad experience across Databricks, Fabric, and Synapse.

Preferred Certifications

Microsoft Fabric Analytics Engineer Associate.

Microsoft Fabric Data Engineer Associate.