Lead Platform Engineer (Databricks)

Retail Insight; Greater London; Hybrid; GBP 110,000 - 140,000; Full time

Benefits offered by this job

Flexible Working – Hybrid work model

Time Off – 25 days annual leave

Learning & Development – Courses on a学

Health & Wellbeing – Private Medical,

Giving Back – Volunteer days & Give as

Extra Perks – Car purchase scheme

Job summary

Retail Insight is seeking a Lead Data Engineer to own the architecture and development of our Databricks environment within the Data & Engineering team. You’ll design scalable data infrastructure behind our products and guide a small team of engineers, shaping platform standards and data governance to enable analytics, ML and AI across the business.

In this senior, hands-on role you’ll mentor engineers, collaborate with Product, Analytics and Data Science, and deliver real-world data

Qualifications

• Proven experience leading the design and delivery of scalable data platforms.
• Hands-on expertise with Databricks, Spark, Delta Lake, Python and SQL.
• Strong understanding of cloud-native data engineering (Azure) and IaC (Terraform, Bicep).
• Expertise in data modelling, ETL/ELT, governance, security and platform operations.
• Experience mentoring engineers and collaborating with stakeholders to deliver high-quality technical solutions.

Responsibilities

• Own and evolve the Databricks platform to be secure, scalable and future-ready.
• Design modern data architectures and standards for analytics, AI and product innovation.
• Lead batch and real-time pipelines using Databricks, Spark and Delta Lake.
• Coach and mentor a team of engineers, promoting technical excellence and continuous improvement.
• Collaborate with Product, Analytics and Data Science to solve business challenges through data.

Skills

Databricks

Spark

Delta Lake

Python

SQL

Cloud-native data engineering

Azure

Terraform

Mentoring engineers

Education

Degree in computer science or data engineering

Tools

dbt

Terraform

Bicep

Job description

Own the platform at the heart of Retail Insight's data strategy

We’re looking for a Lead Data Engineer to join our Data & Engineering team and take ownership of the architecture and development of our Databricks environment.

This is a senior, hands-on role where you’ll design and build the scalable data infrastructure behind our products, while guiding a small team of engineers. You’ll help define our platform standards, data architecture and engineering best practices, creating the foundations for analytics, machine learning and AI across the business.

If you enjoy staying close to the technology, solving complex data challenges and helping other engineers develop, this could be a brilliant opportunity.

What you’ll be doing

• Own and evolve our Databricks platform, making sure it's secure, scalable and built for the future.
• Design modern data architectures and engineering standards that support analytics, AI and product innovation.
• Lead the delivery of batch and real-time data pipelines using Databricks, Spark and Delta Lake.
• Coach and mentor a team of engineers, promoting technical excellence and continuous improvement.
• Work closely with Product, Analytics and Data Science teams to solve real business challenges through data.

Our ideal candidate

• Proven experience leading the design and delivery of scalable, production-grade data platforms.
• Deep hands-on expertise with Databricks, Spark, Delta Lake, Python and SQL.
• Strong understanding of cloud-native data engineering, including Azure and Infrastructure as Code (Terraform, Bicep or similar).
• Expertise in data modelling, ETL/ELT, governance, security, quality and platform operations.
• Experience mentoring engineers and collaborating with stakeholders to deliver high-quality technical solutions.

Even better if you have

• Databricks certifications
• Experience with Unity Catalog, MLflow or Databricks Asset Bundles
• Familiarity with dbt or a similar transformation framework
• Exposure to Kafka, Event Hubs or other real-time streaming technologies
• Experience working with retail or consumer packaged goods data, including point-of-sale, inventory or supply-chain data
• A degree in computer science, data engineering or a related subject, or equivalent practical experience is preferred

Don’t tick every box? That's okay. If the role sounds exciting and you think you could bring something to the team, we'd still love to hear from you.

Some of our extras...

• Flexible Working – Enjoy a hybrid work model (typically 2 days in the office) with flexibility based on business needs, plus a work from anywhere policy to give you freedom to explore.
• Time Off – 25 days annual leave (+ bank holidays), increasing with length of service, plus an extra day off for your birthday! We also operate summer hours so you can make the most of the sunshine.
• Learning & Development – Access a vast range of courses through our learning platform and benefit from structured career progression plans to support your growth.
• Health & Wellbeing – Private Medical Insurance, a healthcare cash plan, and mental health support via Help@Hand. Plus, we'll ensure you have a safe and productive home setup with a workspace assessment.
• Giving Back – Take paid volunteer days to support your local community, donate to your chosen charity through salary sacrifice (we'll match it!), and make a difference with Give as You Earn.
• Extra Perks – A car purchase scheme to make buying a new car easier, plus access to additional benefits through our online platform, including gym and tech discounts.
• Plus much more!

Our interview process

Our interview process will vary slightly depending on the role. We'll always let you know what to expect at each stage and give you plenty of opportunity to ask questions along the way.

People Team Intro Call

An initial conversation with a member of our People Team to learn more about you, your experience and what you’re looking for in your next role. We'll also tell you more about Retail Insight, the opportunity and what you can expect from the interview process.

Skills-Based Interview

This is an opportunity to explore the skills most relevant to the position and understand how you approach your work. This may include a practical task, presentation or technical assessment, and we'll always explain what's involved in advance.

Values-Based Interview

Our final stage focuses on how you work, collaborate and approach different situations, aligned with our values and ways of working at Retail Insight. It's also another opportunity for you to get to know us and make sure Retail Insight feels like the right fit for you.

Be your authentic self

Retail Insight is committed to promoting equal opportunities in employment. All employees and job applicants will receive equal treatment. We actively seek to create an environment where everyone feels respected, supported and encouraged to contribute their best work.