Lead Data Engineer: Azure Fabric, PySpark, Remote + Flexible

CreateFuture; City of Edinburgh; Hybrid; GBP 80,000 - 120,000; Full time

Benefits offered by this job

Flexible working

Hybrid/Remote options

Job summary

CreateFuture in Edinburgh is looking for a Lead Data Engineer to steer the technical direction of a data engagement, working with Microsoft Fabric, Python, and PySpark. You will lead a team of data engineers, define standards for pipelines, architecture, testing, and operations while remaining hands-on with complex challenges.

You will collaborate with client stakeholders and CreateFuture’s engineering teams to ensure delivery is technically excellent and commercially sound.

Qualifications

• Experience designing production data platforms and pipelines.
• Strong SQL, data modelling, and architecture knowledge.
• Proven ability to lead data engineering teams and deliver complex projects.
• Experience with cloud data platforms (Databricks, Snowflake, Synapse) and Azure ecosystem.
• Ability to translate business requirements into a technical roadmap.

Responsibilities

• Own the technical direction of a data engagement, setting standards for pipeline architecture, data modelling, testing, and operational readiness
• Lead and technically direct a team of data engineers, allocating work, reviewing designs and code, and unblocking technical challenges
• Design and evolve data platform architecture on Microsoft Fabric, including data pipelines, Azure OneLake, and the broader Azure ecosystem
• Set and enforce CI/CD, testing, and operational practices across the engagement's data engineering workflows, using Azure DevOps
• Build professional relationships with key client stakeholders, translating business and data requirements into a coherent technical roadmap
• Troubleshoot and resolve the most complex issues in live data pipelines, and lead incident response when things go wrong

Skills

Leadership

Data architecture

Team collaboration

Client communication

Tools

Databricks

Snowflake

Synapse

Microsoft Fabric

Azure OneLake

Azure DevOps

PySpark

Python

Job description

CreateFuture in Edinburgh is looking for a Lead Data Engineer to steer the technical direction of a data engagement, working with Microsoft Fabric, Python, and PySpark. You will lead a team of data engineers, define standards for pipelines, architecture, testing, and operations while remaining hands-on with complex challenges.

You will collaborate with client stakeholders and CreateFuture’s engineering teams to ensure delivery is technically excellent and commercially sound.