Lead Data Engineer (Databricks)

Big Red Recruitment Midlands Ltd; Kirklees; On-site; GBP 68,000 - 83,000; Full time

Job summary

Big Red Recruitment Midlands Ltd is seeking a Lead Data Engineer to drive the transformation of a multi-site UK organisation to a Databricks-based platform. You will own engineering delivery, stay hands-on, design scalable pipelines and establish data layers, while mentoring a small team of Data Engineers.

The role emphasizes building trusted datasets, improving data quality, and shaping engineering standards across the organisation.

Qualifications

• Strong commercial Data Engineering experience.
• Hands-on Databricks expertise.
• Excellent SQL skills.
• Experience designing and developing production data pipelines.
• Strong understanding of modern data-platform architecture.
• Experience working within complex enterprise data environments.
• Previous team leadership or line-management experience.
• Strong stakeholder-management and communication skills.
• The credibility to lead and develop experienced Data Engineers.

Responsibilities

• Lead Data Engineering delivery across Databricks.
• Remaining hands-on with engineering, development and problem-solving.
• Designing and building scalable, reliable data pipelines.
• Helping establish effective bronze, silver and gold data layers.
• Developing curated, trusted and reusable datasets.
• Improving data quality, structure and provenance.
• Supporting the development of enterprise data models and single sources of truth.
• Leading, mentoring and developing a small team of Data Engineers.
• Prioritising and allocating engineering work across multiple projects.
• Establishing strong engineering standards and ways of working.
• Working closely with Data Architecture, Analytics and wider technology teams.
• Acting as a technical sounding board to the Head of Data Engineering.
• Helping reduce manual data processes through better engineering and automation.
• Building internal Databricks knowledge and capability.
• Deputising for senior Data Engineering leadership when required.

Skills

Data Engineering

Databricks

SQL

Python

Job description

Lead Data Engineer - Databricks
Up to £75,000 | Permanent | Huddersfield | 4 days onsite
Want to build the data platform rather than just maintain one?This is an opportunity for a hands‑on Lead Data Engineer to play a key role in a major data transformation - moving a complex, multi-site UK organisation away from fragmented Legacy data and manual processes towards a modern Databricks environment.There's plenty to build, plenty to improve and a genuine opportunity to influence how Data Engineering is done.We're looking for someone who loves Data Engineering, can lead from the front technically and wants to take ownership of a small engineering team while remaining close to the code.

The role
As Lead Data Engineer, you'll take day-to-day ownership of engineering delivery across the Databricks platform, working closely with the Head of Data Engineering and leading a small team of Data Engineers. You'll be joining at an important point in the organisation's transformation. Databricks is becoming the strategic data platform, with a significant programme underway to create better engineered pipelines, trusted data models and reusable data products. This is not a hands-off management role. You'll remain technically involved - designing solutions, building pipelines and solving engineering problems - while providing the direction, standards and leadership the wider team needs.Over time, you'll become a key internal technical authority for Databricks, working closely with Data Architecture and senior data leadership to shape how the platform evolves and what gets built next.

What you'll be doing

• Leading Data Engineering delivery across Databricks
• Remaining hands‑on with engineering, development and problem-solving
• Designing and building scalable, reliable data pipelines
• Helping establish effective bronze, silver and gold data layers
• Developing curated, trusted and reusable datasets
• Improving data quality, structure and provenance
• Supporting the development of enterprise data models and single sources of truth
• Leading, mentoring and developing a small team of Data Engineers
• Prioritising and allocating engineering work across multiple projects
• Establishing strong engineering standards and ways of working
• Working closely with Data Architecture, Analytics and wider technology teams
• Acting as a technical sounding board to the Head of Data Engineering
• Helping reduce manual data processes through better engineering and automation
• Building internal Databricks knowledge and capability
• Deputising for senior Data Engineering leadership when required

The environment
This is a large, complex and highly distributed organisation with a substantial data estate spanning multiple business areas, locations, systems and data sources. There is a significant roadmap of transformation ahead, with opportunities to improve how operational, customer, commercial and product data is brought together and used across the organisation. A major focus is reducing fragmented and manual ways of working and creating properly engineered, trusted data that can support everything from day-to-day decision-making through to advanced analytics, automation and future AI initiatives.You'll be joining while that environment is being built - not once all the interesting decisions have already been made.

What we're looking for

We're looking for a Data Engineer first and a leader second - someone who has progressed into technical leadership but still genuinely enjoys building.

You'll ideally bring:

• Strong commercial Data Engineering experience
• Hands‑on Databricks expertise
• Excellent SQL skills
• Experience designing and developing production data pipelines
• Strong understanding of modern data-platform architecture
• Experience working within complex enterprise data environments
• Previous team leadership or line‑management experience
• The credibility to lead and develop experienced Data Engineers
• Strong stakeholder‑management and communication skills
• The confidence to challenge, prioritise and make decisions
• A proactive, pragmatic and delivery‑focused approach

Python would be useful but isn't essential.
More important is a genuine passion for Data Engineering - making data flow better, removing unnecessary manual processes and building platforms that other teams can confidently use.

Why this one?
Because you're not walking into a finished environment and being asked to maintain it. You'll join while the organisation is actively transforming its data estate, giving you the opportunity to shape the Databricks platform, influence engineering standards, develop the team and see the impact of what you build across a large UK organisation. If you're a hands‑on Lead Data Engineer looking for more ownership, influence and the opportunity to help build something properly, we'd like to hear from you.
Huddersfield - 4 days onsite
Up to £75,000
Permanent