Lead Data Engineer

Xcede; Greater London; Hybrid; GBP 110,000 - 150,000; Full time

Job summary

Xcede is partnering with a leading UK business to hire a Lead Data Engineer in London. You will lead a team of 4 data engineers, setting technical direction on scalable data products and pipelines, with Databricks on AWS at the core.

The role blends hands-on coding with people leadership and close collaboration with data science, analytics and product teams, while shaping a modern cloud data platform and driving data quality.

Qualifications

• Strong commercial Data Engineering experience.
• Previous experience leading, mentoring or managing Data Engineers.
• Strong Python and SQL skills.
• Experience with Databricks on AWS.
• Strong Spark and Airflow experience.
• Experience building scalable data pipelines and integrations.
• Good understanding of CI/CD and Git.
• Experience with Terraform or similar IaC tools.
• Strong understanding of data modelling and data quality.
• A genuine interest in developing people as well as technology.

Responsibilities

• Manage, mentor and develop a team of 4 Data Engineers.
• Provide technical leadership across data engineering projects.
• Design and deliver scalable data products and pipelines.
• Work closely with Data Scientists, Analysts, Architects and Product teams.
• Drive engineering standards around coding, testing and data quality.
• Oversee batch and real-time data integrations.
• Help shape the evolution of a modern cloud data platform.
• Improve engineering processes and ways of working across the wider data community.
• Provide hands-on technical guidance where required.
• Support wider data, ML and AI initiatives.

Skills

Leadership

Python

SQL

Databricks

Spark

Airflow

CI/CD

Git

Terraform

Data modelling

Data quality

Tools

Databricks on AWS

Airflow

Terraform

Job description

Lead Data Engineer | London | 2 days a month in office

I’m working with a leading UK business that is investing heavily in its data and technology capabilities, and they’re looking for a Lead Data Engineer to join the team.

This is a technical leadership role where you’ll specialise in Databricks and manage a team of 4 Data Engineers, while providing technical direction across the wider data engineering function.

The role

• Manage, mentor and develop a team of 4 Data Engineers
• Provide technical leadership across data engineering projects
• Design and deliver scalable data products and pipelines
• Work closely with Data Scientists, Analysts, Architects and Product teams
• Drive engineering standards around coding, testing and data quality
• Oversee batch and real-time data integrations
• Help shape the evolution of a modern cloud data platform
• Improve engineering processes and ways of working across the wider data community
• Provide hands-on technical guidance where required
• Support wider data, ML and AI initiatives

What they’re looking for

• Strong commercial Data Engineering experience
• Previous experience leading, mentoring or managing Data Engineers
• Strong Python and SQL skills
• Experience with Databricks on AWS
• Strong Spark and Airflow experience
• Experience building scalable data pipelines and integrations
• Good understanding of CI/CD and Git
• Experience with Terraform or similar IaC tools
• Strong understanding of data modelling and data quality
• A genuine interest in developing people as well as technology

If you’re a Data Engineer who has moved into technical leadership or people management and want to lead a team while staying close to the technology, get in touch.