Lead Data Engineer - Data Platform & Pipelines (Remote)

CreateFuture; Greater London; Hybrid; GBP 90,000 - 120,000; Full time

Job summary

CreateFuture in the United Kingdom seeks a Lead Data Engineer to own the technical direction of a data engagement, working with Microsoft Fabric, Python, and PySpark. You will lead a team of data engineers, set standards for pipelines and architecture, and ensure delivery is technically excellent and commercially sound.

You will design data platform architecture across Microsoft Fabric, Azure OneLake, and the Azure ecosystem, while enforcing CI/CD and operational practices.

Qualifications

• Extensive hands-on experience with cloud data platforms such as Microsoft Fabric, Databricks, Snowflake or Synapse.
• Strong SQL skills and data modelling knowledge (Kimball, data vault, lakehouse patterns).
• Proficiency in Python and PySpark for batch data processing at scale.

Responsibilities

• Own the technical direction of a data engagement, setting standards for pipeline architecture, data modelling, testing, and operational readiness.
• Lead and technically direct a team of data engineers, allocating work, reviewing designs and code, and unblocking technical challenges.
• Design and evolve data platform architecture on Microsoft Fabric, including data pipelines, Azure OneLake, and the broader Azure ecosystem.
• Set and enforce CI/CD, testing, and operational practices across the engagement's data engineering workflows, using Azure DevOps.
• Build professional relationships with key client stakeholders, translating business and data requirements into a coherent technical roadmap.
• Troubleshoot and resolve the most complex issues in live data pipelines, and lead incident response when things go wrong.

Skills

SQL expertise

Python

PySpark

Team leadership

CI/CD practices

Tools

Microsoft Fabric

Databricks

Snowflake

Synapse

Azure DevOps

Azure OneLake

Job description

CreateFuture in the United Kingdom seeks a Lead Data Engineer to own the technical direction of a data engagement, working with Microsoft Fabric, Python, and PySpark. You will lead a team of data engineers, set standards for pipelines and architecture, and ensure delivery is technically excellent and commercially sound.

You will design data platform architecture across Microsoft Fabric, Azure OneLake, and the Azure ecosystem, while enforcing CI/CD and operational practices.