Senior Data Engineer

Formula.; West of England; Hybrid; GBP 88,000 - 115,000; Full time

Job summary

Formula. in Bristol is building a new data platform on Microsoft Azure and seeks a hands‑on Databricks specialist to stand up the platform and publish pipelines to a gold layer in Azure SQL.

The role focuses on end‑to‑end data pipelines, Unity Catalog, Delta Lake, and CDC ingestion. You’ll be in the office at first and then hybrid, with a focus on robust, production‑grade delivery.

Qualifications

• 6+ years in data engineering with 3+ years running production pipelines on Databricks.
• Experience taking a Databricks platform from empty workspace to reliable production.
• Deep knowledge of Unity Catalog, Delta Lake, PySpark and Spark SQL.

Responsibilities

• Stand up Azure Databricks as code: workspaces, Unity Catalog, permissions and environments.
• Build reusable config‑driven ingestion and data quality frameworks.
• Deliver end‑to‑end pipelines from source systems to a gold model in Azure SQL.
• Implement CDC ingestion with full history.
• Set engineering standards: CI/CD, testing, environment promotion, scheduling and alerting.
• Make the platform production‑ready with runbooks, backfill, replay, governance and cost control.
• Support go‑lives and stabilise pipelines after launch.
• Leverage AI‑assisted development with human review.
• Coach engineers to run pipelines confidently.

Skills

Databricks

Unity Catalog

Delta Lake

PySpark

Spark SQL

CDC

CI/CD

Testing

Environment promotion

Go‑live support

Tools

Lakeflow Connect

Auto Loader

Lakebase

Terraform

ADLS

Key Vault

Entra ID

Job description

Bristol | Office-first, then hybrid

Initial 3 months, likely to extend | Up to £550 per day, outside IR35

Help build a new data platform from the ground up.

My client is growing its technology team and building a SaaS data platform on Microsoft Azure for enterprise customers. The team is small, based in Bristol and moves fast. They need a hands‑on Databricks specialist to stand up the platform, get the first pipelines live and set the patterns that future work will build on.

What you’ll be doing

• Standing up Azure Databricks as code: workspaces, Unity Catalog, permissions and environments
• Building reusable, config‑driven frameworks for ingestion, medallion layers, data quality and publishing
• Delivering pipelines end to end, from source systems through to a modelled gold layer published to Azure SQL
• Implementing CDC ingestion with full history
• Setting engineering standards: CI/CD, testing, environment promotion, scheduling and alerting
• Making the platform production‑ready: runbooks, backfill and replay, recovery, governance, lineage and cost control
• Supporting go‑lives and stabilising pipelines after launch
• Using AI‑assisted delivery with strong human review
• Coaching the engineers around you so they can run pipelines confidently without you.

What you’ll bring

• 6+ years in data engineering, including 3+ years running production pipelines on Databricks
• Experience taking a Databricks platform from an empty workspace to reliable production
• Deep, current knowledge of Unity Catalog, Asset Bundles, Delta Lake (MERGE and incremental loads), PySpark, Spark SQL, Jobs and serverless compute
• Strong ingestion experience: Lakeflow Connect or equivalent CDC from SQL Server, Auto Loader, file‑based sources and SCD Type 2
• A track record of building frameworks that other engineers adopted
• Proven use of AI‑assisted development on production work, with good judgement about where the tools fall short
• A preference for working in person, and the ability to be in the Bristol office at the start and regularly after that
• Eligibility for UK SC clearance

Nice to have:

Databricks Data Engineer Professional certification, Lakeflow Declarative Pipelines or Lakebase, Terraform, ADLS, Key Vault and Entra ID, data contracts or semantic layers, or current SC clearance.

Why this role? You’ll shape how the platform is built from day one, and your work will be used long after the contract ends.