Databricks SME

iO Associates; Greater London; On-site; GBP 111,000 - 157,000; Full time

Job summary

iO Associates is seeking a Databricks SME to join a growing business and optimise an existing data platform. You will improve performance, governance and architecture as part of a wider greenfield initiative.

You’ll work with stakeholders to review the current setup, identify gaps and re-architect where needed, helping the business scale with Databricks.

Qualifications

• Proven experience as a Databricks SME with hands-on platform management.
• Strong data architecture, pipelines, and AI/data foundations.
• Excellent communication and ability to guide discovery and planning.
• Retail industry experience is advantageous.

Responsibilities

• Review current Databricks setup and identify gaps.
• Re-architect to improve performance, governance and scale.
• Collaborate with stakeholders to align with business goals.

Skills

Stakeholder collaboration

Technical leadership

Independent work

Retail data experience

Tools

Databricks

Delta Lake

Python / PySpark

SQL

CI/CD

Terraform

Job description

Join Our Client as a Databricks SME – Strategic Technology Leadership Opportunity

A growing business is investing further in its data platform — but this one’s a little different. Rather than building from scratch, they need a genuine Databricks SME to come in and optimise an existing platform: improving performance, cost efficiency, governance, and architecture as part of a wider greenfield initiative.

This is a high-impact contract for someone who knows Databricks inside out and wants to make an immediate, visible difference — not just maintain what’s already there.

You’ll work closely with stakeholders, reviewing the current setup, identifying where it’s falling short, and re-architecting where needed — all while the business scales.

• Proven experience as a Databricks SME, with hands‑on expertise in implementing and managing Databricks platforms.
• Strong understanding of data architecture, data pipelines, and AI/data foundations.
• Familiarity with Atlassian tools and their integration within complex data environments.
• Excellent communication skills, capable of guiding teams through technical discovery and strategic planning.
• Ability to work independently and collaboratively in a fast‑paced, innovative environment.
• Prior experience in the retail industry or similar sectors is advantageous.

Tech:

• Databricks (Unity Catalog, cluster & job optimisation)
• Delta Lake / Lakehouse architecture
• Python / PySpark
• SQL
• CI/CD, Terraform, orchestration tool if relevant

Bonus points for:

cost governance frameworks, data quality/observability tooling, or experience taking a platform from “it works” to “it scales.”

If you’ve ever looked at a Databricks environment and thought “I know exactly how I’d fix this,” this is that project.