Lead Data Engineer

Matchtech; Richmond; On-site; GBP 65,000 - 75,000; Full time

Job summary

Matchtech is seeking a Lead Data Engineer (Databricks) to own the architecture and development of our Databricks environment. You will lead a small team, define platform standards and support analytics, ML and AI initiatives.

The role is senior and hands-on, focused on scalable data infrastructure and collaboration with data science and product teams. Remote or Hybrid UK with occasional Richmond visits.

Qualifications

• Senior or lead data engineer with production data platform experience.
• Deep hands-on Databricks, Spark and Delta Lake expertise.
• Proficiency in Python and SQL.
• IaC experience: Terraform, Bicep or CloudFormation.
• Cloud data platforms, preferably Azure, with data factory and storage.
• Strong data modelling, ETL/ELT, and governance knowledge.
• Experience leading/mentoring engineers and stakeholder management.
• Databricks certification is a plus.

Responsibilities

• Lead design, development and optimisation of a Databricks-based data platform.
• Define enterprise-scale data architectures on Databricks and cloud platforms.
• Establish lakehouse standards, data modelling and governance.

Skills

Databricks

Spark

Delta Lake

Python

SQL

Cloud platforms

Terraform

Bicep

CloudFormation

Data modelling

ETL/ELT

Security governance

Mentoring

Education

Bachelor's degree in CS or Data Eng

Tools

Databricks Workflows

Azure Data Factory

ADLS Gen2

Azure DevOps

Job description

Lead Data Engineer (Databricks)
Salary £65,000
Type Full time, permanent
Hours 37.5 per week
Location Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)

Overview

We are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.

Key responsibilities

* Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability
* Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP
* Define lakehouse standards, data modelling approaches and governance frameworks
* Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies
* Establish security, access control, data quality and compliance standards
* Optimise performance, scalability and cloud cost management
* Lead and mentor a small team of data engineers, setting technical standards and reviewing code
* Partner with data science, analytics and product teams to ensure the platform supports their work
* Own data quality, governance and observability across pipelines
* Support AI and ML enablement including feature stores and model deployment patterns
* Translate business requirements into technical solutions
* Contribute to platform roadmap, tooling selection and continuous improvement

Required experience

* Strong experience as a senior or lead data engineer building and scaling production data platforms
* Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows
* Python and SQL proficiency
* Infrastructure as code experience using Terraform, Bicep or CloudFormation
* Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps
* Strong understanding of data modelling, ETL and ELT patterns, distributed processing
* Experience with security, governance and data platform operations
* Experience leading or mentoring engineers, with strong stakeholder management
* Rigorous approach to data quality and testing
* Databricks certification

Desirable

* Unity Catalog, MLflow and Databricks Asset Bundles
* dbt or similar transformation frameworks
* Retail or CPG data such as POS, inventory, supply chain
* Streaming technologies such as Kafka or Event Hubs

Education

* Degree in computer science, data engineering or related field, or equivalent practical experience