Lead Data Engineer

Big Red Recruitment; Stratford-upon-Avon; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A recruitment agency specializing in data roles is seeking an experienced Data Engineer for a hybrid position. The successful candidate will lead the design and implementation of a new Databricks-based data warehousing solution, utilizing their expertise in PySpark and providing training to the team. This role offers a salary range of £60,000 to £80,000 depending on experience, along with the opportunity to impact modern data platforms significantly.

Qualifications

• Proven experience with Azure Databricks and PySpark.
• Experience in data warehousing architectures.
• Strong knowledge in designing data models and ETL pipelines.

Responsibilities

• Lead the design and implementation of a new data warehousing solution.
• Monitor, tune, and optimize data platforms for performance.
• Upskill team members in Databricks best practices.

Skills

Azure Databricks

PySpark

Data modelling

ETL pipeline development

Performance tuning

Job description

Are you a skilled Data Engineer looking to step into the world of architecture?

An exciting opportunity has opened for an experienced Data Engineer to join our national Data & Analytics function at a time of significant technical modernisation.

The team is about to embark on a greenfield project to build a future‑proof data warehousing platform using Azure Data Factory and Databricks, with a clear focus on scalability, quality, and best practice.

Working within a specialist Data Platform & Engineering team, you’ll join as the expert within Azure Databricks and PySpark. You'll help to upskill our teams' knowledge of Databricks, get involved in data modelling and ETL pipeline development, integrations, and performance tuning. This is an ever‑evolving project as the business becomes more and more data‑driven and your knowledge of Databricks will be pivotal in shaping the architecture.

This is an ideal role for a Lead Data Engineer ready to step into an architectural role, or an established architect keen to take ownership of a highly visible and strategic data platform project.

What you’ll be doing:

• Leading the design and implementation of a new Databricks‑based data warehousing solution
• Designing and developing data models, ETL pipelines, and data integration processes
• Large‑scale data processing using PySpark
• Monitoring, tuning, and optimising data platforms for reliability and performance
• Upskilling the wider team in Databricks best practices, including modern architecture patterns

Location: Atherstone (Hybrid – 3 days office / 2 days remote)

Salary: £60 000 to £80 000 depending on experience

If you're passionate about shaping modern data platforms and have the technical expertise to make a measurable impact, we’d love to hear from you.

Apply now as we have interview slots available!

Please note: Visa sponsorship is not provided for this role and we are only considering applications of candidates who have permanent residency in the UK.

We are an equal opportunity recruitment company. This means we welcome applications from all suitably qualified people regardless of race, sex, disability, religion, sexual orientation or age.

We are particularly invested in Neurodiversity inclusion and offer reasonable adjustments in the interview process. Reasonable adjustments are changes that we can make in the interview process if your disability puts you at a disadvantage compared with others who are not disabled. If you would benefit from a reasonable adjustment in your interview process, please call or email one of our recruiters.