Data Engineer Lead

Matchtech; England; On-site; GBP 70,000 - 90,000; Full time

Job summary

A leading technology firm in the UK is seeking an experienced Data Engineer Lead to oversee the design and delivery of data solutions. The ideal candidate will have significant expertise in Databricks and Azure data services, along with strong leadership abilities in managing data engineering teams. This contract role offers opportunities for career development in a dynamic environment.

Qualifications

• Experience with Databricks, delivering curated silver and gold data layers.
• Proven experience leading data engineering teams and managing sprint ceremonies.
• Excellent written and verbal communication skills.

Responsibilities

• Lead the Conform team responsible for building and maintaining pipelines.
• Act as the main liaison with product owners and stakeholders.
• Oversee sprint ceremonies ensuring backlog prioritisation.

Skills

Databricks

Azure data services

Python

SQL

PySpark

Stakeholder management

Tools

Azure Service Bus

Azure Logic Apps

Azure DevOps

Azure Data Factory

Job description

Our client, a leading organisation in the technology sector, is currently seeking a Data Engineer Lead to join their team on a contract basis. This role involves leading the design and delivery of the National Energy System Operator (NESO) for the Conform data layer, providing trusted, business-ready data across the enterprise. The ideal candidate will have deep expertise in Databricks and Azure data services.

Key Responsibilities:

• Lead the Conform team responsible for building and maintaining pipelines from bronze through silver to gold layers in Databricks.
• Act as the main liaison with product owners and stakeholders, managing ad hoc requests, escalations, and shifting priorities.
• Oversee sprint ceremonies including refinement, reviews, retrospectives, and PI planning, ensuring clear backlog prioritisation and achievable commitments.
• Track and report on sprint KPIs such as velocity, story points, and successful delivery of demos.
• Chair combined stand-ups with data product and test teams to align dependencies and resolve issues quickly.
• Collaborate with acquisition teams to ensure upstream data ingestion supports downstream transformations.
• Partner with QA vendors to validate Conform outputs, manage testing dependencies, and resolve bugs.
• Safeguard the team from misplaced accountability by ensuring transparent communication, issue analysis, and resolution.
• Mentor Databricks engineers, providing technical guidance and preparing team members for future leadership.
• Support cross-company collaboration by coordinating with multiple vendors and maintaining high-quality delivery standards.

Job Requirements:

• Experience with Databricks, delivering curated silver and gold data layers.
• Azure Service Bus and Event Hub
• Azure Logic Apps and Functions
• Azure DevOps for agile delivery and sprint tracking
• Azure Data Factory (nice to have)
• Proficiency in Python, SQL, PySpark; Foundry nice to have
• Proven experience leading data engineering teams, managing sprint ceremonies, and coordinating delivery
• Strong stakeholder management skills, with ability to manage prioritisation and escalations across multiple business units
• Familiarity with PI planning and agile delivery cycles
• Excellent written and verbal communication, with ability to present progress and defend technical decisions

If you are an experienced Data Engineer Lead looking for a new opportunity to further develop your career in the technology sector, we would love to hear from you. Apply now to join our client's dynamic and talented team.

Matchtech is a STEM Recruitment Specialist, with over 40 years’ experience