Senior Data Engineer

Primus Connect; United States; Remote; GBP 96,000 - 126,000; Full time

Benefits offered by this job

Fully Remote

Job summary

Primus Connect is seeking a Senior Data Engineer for a 2-month fully remote contract in the UK. You will migrate 72 T-SQL stored procedures from Azure Synapse Dedicated SQL Pool to Databricks, ensuring performance and data accuracy. Strong expertise in Synapse, Databricks, Spark SQL, PySpark, and Delta Lake is essential.

The role focuses on optimizing migration patterns, validating results, and delivering a production-ready solution by end of December 2026. Start date 1 November 2026.

Qualifications

• Hands-on experience migrating complex SQL workloads from Synapse Dedicated SQL Pools to Databricks.
• Experience migrating SQL workloads between MPP SQL environments and Databricks.
• Strong understanding of architecture differences between Synapse SQL and Databricks.

Responsibilities

• Migrate ~72 T-SQL stored procedures from Azure Synapse to Databricks.
• Analyse code dependencies and execution patterns for migration approach.
• Refactor T-SQL logic for Databricks SQL, Spark SQL, and PySpark.
• Ensure datasets and pipelines are available for business consumption.
• Address differences between Synapse and Databricks execution models.
• Optimize query execution and data transformations.
• Validate data accuracy against migration thresholds.
• Troubleshoot migration issues and deliver production-ready solution.

Skills

Azure Synapse

Databricks

T-SQL

PySpark

Spark SQL

Delta Lake

Lakehouse

MPP architecture

Query tuning

Data migration

Tools

Databricks SQL

Azure Synapse Analytics

Job description

£600 per day | Outside IR35 | Fully Remote | 2-Month Contract | Start: 1st November

We are looking for a highly experienced Senior Data Engineer to support a critical data platform migration project for one of our clients.

This is an initial 2-month contract, working fully remotely, with a target delivery date of the end of December.

The successful consultant will be responsible for migrating an existing Azure Synapse Dedicated SQL Pool codebase into Databricks, ensuring the migrated solution delivers equivalent or improved performance, maintains data accuracy and supports ongoing business consumption.

We are specifically looking for someone with hands‑on experience migrating complex SQL workloads from Synapse Dedicated SQL Pools to Databricks, rather than general Databricks engineering experience alone.

Key Responsibilities

• Migrate approximately 72 T-SQL stored procedures from Azure Synapse Dedicated SQL Pool into Databricks.
• Analyse existing SQL code, dependencies and execution patterns to determine the most effective migration approach.
• Refactor and optimise complex T-SQL logic for Databricks SQL, Spark SQL and/or PySpark.
• Identify, remove and rework dependencies on existing Claims dimensional model fact and dimension tables.
• Ensure migrated datasets and pipelines are available for downstream business consumption.
• Address architectural differences between Synapse's MPP SQL processing engine and Databricks' Spark-based execution.
• Optimise query execution, joins, data transformations and pipeline performance.
• Deliver equivalent or improved pipeline performance following migration.
• Perform comprehensive reconciliation and validation to ensure data accuracy and consistency with existing migration thresholds.
• Troubleshoot migration issues and deliver a production-ready solution within the agreed timeline.
• Strong hands‑on experience with Azure Synapse Analytics, specifically Dedicated SQL Pools.
• Advanced T‑SQL skills, including complex stored procedures, transformations and query optimisation.
• Deep understanding of MPP architecture, distributed tables, execution plans and performance tuning.
• Extensive commercial experience with Azure Databricks, Spark SQL, Databricks SQL and PySpark.
• Strong understanding of Apache Spark execution, data distribution, joins, shuffles and performance optimisation.
• Experience with Delta Lake and modern lakehouse architecture.

SQL / Synapse to Databricks Migration

• Demonstrable experience migrating SQL workloads from Synapse Dedicated SQL Pools or similar MPP SQL environments into Databricks.
• Strong understanding of the architectural and processing differences between Synapse SQL and Databricks.
• Experience converting complex T‑SQL code into scalable, maintainable Databricks solutions.
• Ability to identify and resolve SQL compatibility, data transformation and performance challenges.

Data Quality & Validation

• Experience reconciling source and target datasets following complex migrations.
• Strong understanding of data validation, regression testing and data quality.
• Ability to work independently and take ownership of delivery.
• Previous experience migrating large volumes of SQL stored procedures.
• Experience working with dimensional data models, including fact and dimension tables.
• Databricks Workflows, orchestration and performance tuning.
• Experience delivering time-critical data platform migration projects.
• Role: Senior Azure Synapse to Databricks Migration Consultant
• Rate: Up to £600 per day
• Location: Fully Remote
• Duration: Initial 2 months
• Start Date: 1st November 2026
• Target Delivery: End of December 2026