Senior Data Engineer

Consult

England

Hybrid

GBP 96,000 - 126,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Fully remote

Job summary

Consult is seeking a Senior Data Engineer for a fully remote, 2-month contract to migrate an Azure Synapse Dedicated SQL Pool codebase to Databricks, delivering parity or improved performance and ensuring data accuracy.

You will refactor ~72 T-SQL procedures, optimise Spark/PySpark queries and validate datasets for downstream consumption, addressing architecture differences between Synapse and Databricks.

Qualifications

  • Hands-on experience with Azure Synapse Dedicated SQL Pool migrations.
  • Advanced T-SQL skills including complex stored procedures and optimization.
  • Experience migrating large SQL workloads to Databricks.
  • Knowledge of Delta Lake and lakehouse architectures.

Responsibilities

  • Migrate ~72 T-SQL stored procedures from Synapse to Databricks.
  • Analyse code, dependencies and execution patterns for migration strategy.
  • Refactor T-SQL logic for Databricks SQL, Spark SQL and PySpark.
  • Ensure datasets and pipelines are available for downstream consumption.

Skills

Azure Synapse Analytics
Databricks & Spark SQL
T-SQL & SQL migration
PySpark
Delta Lake

Tools

Azure Databricks
Spark SQL

Job description

Senior Data Engineer – Contract

£600 per day | Outside IR35 | Fully Remote | 2-Month Contract | Start: 1st November

We are looking for a highly experienced Senior Data Engineer to support a critical data platform migration project for one of our clients.

This is an initial 2-month contract, working fully remotely, with a target delivery date of the end of December.

The successful consultant will be responsible for migrating an existing Azure Synapse Dedicated SQL Pool codebase into Databricks, ensuring the migrated solution delivers equivalent or improved performance, maintains data accuracy and supports ongoing business consumption.

We are specifically looking for someone with hands-on experience migrating complex SQL workloads from Synapse Dedicated SQL Pools to Databricks, rather than general Databricks engineering experience alone.

Key Responsibilities
  • Migrate approximately 72 T-SQL stored procedures from Azure Synapse Dedicated SQL Pool into Databricks.
  • Analyse existing SQL code, dependencies and execution patterns to determine the most effective migration approach.
  • Refactor and optimise complex T-SQL logic for Databricks SQL, Spark SQL and/or PySpark.
  • Identify, remove and rework dependencies on existing Claims dimensional model fact and dimension tables.
  • Ensure migrated datasets and pipelines are available for downstream business consumption.
  • Address architectural differences between Synapse's MPP SQL processing engine and Databricks' Spark-based execution.
  • Optimise query execution, joins, data transformations and pipeline performance.
  • Deliver equivalent or improved pipeline performance following migration.
  • Perform comprehensive reconciliation and validation to ensure data accuracy and consistency with existing migration thresholds.
  • Troubleshoot migration issues and deliver a production-ready solution within the agreed timeline.
Essential Skills & Experience
Azure Synapse Dedicated SQL Pool
  • Strong hands-on experience with Azure Synapse Analytics, specifically Dedicated SQL Pools.
  • Advanced T-SQL skills, including complex stored procedures, transformations and query optimisation.
  • Deep understanding of MPP architecture, distributed tables, execution plans and performance tuning.
Databricks & Apache Spark
  • Extensive commercial experience with Azure Databricks, Spark SQL, Databricks SQL and PySpark.
  • Strong understanding of Apache Spark execution, data distribution, joins, shuffles and performance optimisation.
  • Experience with Delta Lake and modern lakehouse architecture.
SQL / Synapse to Databricks Migration
  • Demonstrable experience migrating SQL workloads from Synapse Dedicated SQL Pools or similar MPP SQL environments into Databricks.
  • Strong understanding of the architectural and processing differences between Synapse SQL and Databricks.
  • Experience converting complex T-SQL code into scalable, maintainable Databricks solutions.
  • Ability to identify and resolve SQL compatibility, data transformation and performance challenges.
Data Quality & Validation
  • Experience reconciling source and target datasets following complex migrations.
  • Strong understanding of data validation, regression testing and data quality.
  • Ability to work independently and take ownership of delivery.
Desirable Experience
  • Previous experience migrating large volumes of SQL stored procedures.
  • Insurance, Claims or financial services data experience.
  • Experience working with dimensional data models, including fact and dimension tables.
  • Databricks Workflows, orchestration and performance tuning.
  • Experience delivering time-critical data platform migration projects.
Contract Details
  • Role: Senior Azure Synapse to Databricks Migration Consultant
  • Rate: Up to £600 per day
  • IR35: Outside IR35
  • Location: Fully Remote
  • Duration: Initial 2 months
  • Start Date: 1st November 2026
  • Target Delivery: End of December 2026
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer: Synapse to Databricks (Remote)
Senior Data Engineer: Synapse to Databricks (Remote)

Consult • England

Hybrid
GBP 96,000 - 126,000
Fully remote
Senior Data Engineer - Contract Role
Senior Data Engineer - Contract Role

La Fosse • Milton Keynes

Hybrid
GBP 92,250 - 96,862
Senior Databricks Engineer (Contract, £550-600 p/d)
Senior Databricks Engineer (Contract, £550-600 p/d)

Layke AI • Birmingham

Hybrid
GBP 60,000 - 80,000
Senior Data Engineer
Senior Data Engineer

GCS Recruitment • Greater London

Hybrid
GBP 136,000 - 178,000
Databricks Engineer
Databricks Engineer

TechShack • United Kingdom

Remote
GBP 101,475 - 119,925
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Tenth Revolution Group • United Kingdom

Remote
GBP 90,000 - 130,000
Senior Data Engineer – Azure Databricks
Senior Data Engineer – Azure Databricks

Talensa Partners • Greater London

Hybrid
GBP 90,000 - 130,000
Competitive Salary
Annual Bonus and Growth Share Scheme
Pension
+3
Data Engineer – Databricks – Contract Role
Data Engineer – Databricks – Contract Role

Data & AI • Milton Keynes

Hybrid
GBP 92,000 - 101,000
Senior Data Engineer (Remote)
Senior Data Engineer (Remote)

Ceox Services Ltd • United Kingdom

Remote
GBP 92,000 - 129,000
Senior Data Engineer
Senior Data Engineer

Nicoll Curtin • Leicester

Hybrid
GBP 95,755 - 125,644
Outside IR35
Modern tech stack
Friendly team