Lead Data Architect

Collage Recruitment; Greater London; Hybrid; GBP 90,000 - 140,000; Full time

Job summary

Collage Recruitment advertises a Lead Data Architect for Insurance Data & Analytics in London. The role centers on migrating a legacy benchmarking engine to a modern Databricks Lakehouse and building product tooling on top of it.

You will own the UK data platform, drive CI/CD and IaC, and mentor engineers while collaborating with global teams on platform standards. This is a senior leadership technical role with direct impact on data products.

Qualifications

• Experience delivering Databricks migration at production scale.
• Strong skills in Python, PySpark and SQL.
• Track record migrating SQL Server workloads to cloud-native platforms.

Responsibilities

• Lead migration of legacy SQL to Databricks pipelines and orchestrations.
• Define target-state data architecture (Lakehouse, Medallion).
• Codify canonical metric logic with testable code.
• Own Unity Catalog governance: schema, tables, access control.

Skills

Databricks migration

Python

PySpark

SQL

Stakeholder management

CI/CD

Infrastructure as Code

Performance tuning

Data governance

Tools

Databricks

Unity Catalog

dbt

Job description

Lead Data Architect - Insurance Data & Analytics (Databricks Migration)

London (Hybrid) |Permanent, Full-Time

About the Role

Our client, a leading provider of benchmark data and analytics for the financial services sector, is hiring the first senior engineering leader for their UK Insurance data business. This is a newly created role, not a backfill.

This is a rare opportunity to own a platform transformation from the ground up: migrating a legacy SQL benchmarking engine onto a modern Databricks Lakehouse, and then building the product tooling on top of it.

Our client provides benchmark data behind pricing, acquisition, renewal, and retention decisions across the UK insurance market, built on a proprietary, behavioural dataset that underpins trusted relationships with senior commercial decision-makers across the industry.

You'll report directly to the Head of Insurance, based in London, and will be the senior technical voice for the UK Insurance business - with the option to line-manage an existing mid-level engineer as the team grows. You'll also work closely with a broader, global engineering function on platform standards, while owning delivery locally.

What You'll Do

• Lead the migration of legacy SQL solutions into Databricks Jobs, Workflows, and PySpark/SQL pipelines, starting with a lift-and-shift of core benchmark data, while maintaining operational SLAs throughout
• Define the target-state data architecture (Lakehouse, Medallion, domain-oriented data products) to support reporting, analytics, and future AI/ML use cases
• Work directly with business leadership and analysts to codify canonical metric logic - replacing spreadsheet calculations with documented, testable code
• Own UK Unity Catalog governance: catalogue, schema, table design, ownership, and access control
• Build automated QA and reconciliation between legacy and new outputs, with the observability to give internal and external stakeholders confidence in the data
• Implement CI/CD and Infrastructure as Code for Databricks assets
• Tune Spark and SQL workloads for performance and cost
• Act as the primary link between UK delivery and the centralised platform engineering function
• Provide technical leadership and mentorship, with the option to line-manage a mid-level engineer

What We're Looking For

• Hands-on experience delivering a Databricks migration at production scale - this is the core of the role
• Strong stakeholder and relationship management skills. You'll work closely with business leadership, analysts, and engineers across regions, often with ambiguity to navigate
• Advanced Python, PySpark, and SQL
• A track record migrating SQL Server workloads, stored procedures, and scheduled jobs to modern, cloud-native platforms
• Solid understanding of Unity Catalog governance, security models, and access control
• Experience with CI/CD, automated testing, and Infrastructure as Code

We'd love if you also have:

• Experience in a smaller, B2B-focused fintech environment - this role suits someone used to close commercial context and hands-on delivery, rather than large enterprise-scale environments with extensive legacy system sprawl
• Insurance, financial services, pricing, or marketing analytics background
• Familiarity with UK GDPR, data residency, and regional governance
• dbt or analytics engineering experience

Why This Role

This isn't a startup. It's an established, well-regarded business in its space, but you'll have the ownership and autonomy of building something from scratch. You'll shape the architecture, the standards, and potentially the team, with the backing and technical standards of a wider global engineering organisation behind you.