Head of Data Engineering

TechYard; Leeds; On-site; GBP 110,000 - 170,000; Full time

Job summary

TechYard in the United Kingdom seeks an experienced Head of Data Engineering to lead and grow a data engineering function within a major retail business. This senior role shapes strategy, oversees delivery, and drives best practices across an Azure Databricks environment, including governance and cost management.

You will lead internal teams and external delivery partners, establish DataOps standards, and advance data quality and platform performance while collaborating with data science and

Qualifications

• Senior leadership experience in data engineering environments.
• Proven track record delivering enterprise-scale data platforms in the cloud.
• Deep knowledge of data governance, quality and DataOps practices.

Responsibilities

• Lead and develop the data engineering function, including internal teams and external delivery partners.
• Own the design, development and delivery of enterprise-scale data products.
• Establish and embed engineering standards, best practices and a robust DataOps operating model.
• Drive improvements across data quality, governance, monitoring and platform performance.
• Oversee data engineering costs, optimisation and delivery efficiency.
• Work closely with senior stakeholders, data science and technology teams.
• Shape the future direction of data applications, ML and AI engineering.
• Proven experience leading enterprise-scale data engineering teams.

Skills

Leadership

Data architecture

Data governance

DataOps

Performance optimization

Cost management

Stakeholder mgmt

Azure Databricks

CI/CD

Infrastructure as Code

ML/AI exposure

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Delta Live Tables

Lakeflow

MLflow

Job description

We're looking for an experienced Head of Data Engineering to lead and develop a growing data engineering function within a major retail business.

This is a senior leadership position, responsible for shaping the data engineering strategy, overseeing delivery and establishing best-practice engineering standards across an enterprise-scale Azure Databricks environment.

You'll lead internal engineering teams and third-party partners, driving improvements across data architecture, DataOps, governance, quality and cost efficiency.

Key Responsibilities:

• Lead and develop the data engineering function, including internal teams and external delivery partners.
• Own the design, development and delivery of enterprise-scale data products.
• Establish and embed engineering standards, best practices and a robust DataOps operating model.
• Drive improvements across data quality, governance, monitoring and platform performance.
• Oversee data engineering costs, optimisation and delivery efficiency.
• Work closely with senior stakeholders, data science and technology teams.
• Help shape the future direction of data applications, machine learning and AI engineering.
• Extensive experience leading enterprise-scale data engineering teams.
• Strong hands-on experience with Azure Databricks and the wider Azure ecosystem.
• Strong knowledge of Unity Catalog, Delta Lake, Lakeflow / Delta Live Tables and Databricks Workflows.
• Experience implementing CI/CD, Infrastructure as Code and automated testing.
• Proven experience establishing DataOps practices, engineering standards and data governance.
• Strong understanding of data architecture, performance optimisation and cost management.
• Experience managing third-party delivery partners and engaging with senior stakeholders.
• Experience building data applications on the Databricks Lakehouse.
• Exposure to MLflow, Mosaic AI, RAG and agentic AI solutions.
• Experience taking machine learning and AI workloads into production.
• Background in retail, grocery, supply chain or another high-volume transactional environment.

This is a fantastic opportunity for an established data engineering leader to take ownership of a significant enterprise data environment and help shape its future direction, including its growing AI capabilities.