Senior Databricks Data Engineer

McCabe & Barton; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

McCabe & Barton is seeking a contract Data Migration Specialist in the Greater London area to transform legacy SQL Server data solutions into a modern Databricks architecture. You will assess the current landscape, map dependencies, and redesign critical structures while migrating selected workloads to a scalable data platform.

You will collaborate with business, accounting, and technology stakeholders to ensure successful delivery, with emphasis on data quality, lineage, and robust

Qualifications

• Extensive experience with SQL Server, T‑SQL, Databricks, and Spark SQL development.
• Strong expertise in data modelling, with a solid understanding of Medallion Architecture.
• Proven track record of migrating SQL Server, SSIS, stored procedures, ETL processes, and reporting workloads to Databricks or similar cloud‑based data platforms.
• Experience with data profiling, metadata management, lineage analysis, dependency mapping, source‑to‑target mappings, and data quality controls.
• Ability to assess complex legacy systems and determine the most effective approach for migration, redesign, consolidation, replacement, or retirement.
• Excellent documentation, stakeholder engagement, communication, and knowledge transfer skills.

Responsibilities

• Conduct a detailed inventory of the current data environment, including databases, schemas, tables, views, stored procedures, SQL Agent jobs, SSIS packages, feeds, reports, consumers, permissions, and operational processes.
• Document data ownership, lineage, business keys, relationships, calculations, transformations, refresh frequencies, and downstream dependencies.
• Perform data profiling and quality assessments, including completeness, accuracy, duplication, referential integrity, timeliness, and historical coverage.
• Collaborate with business and technical SMEs to determine which data structures, calculations, and reports should be retained, redesigned, consolidated, enhanced, or retired.
• Convert, redesign, and migrate T‑SQL code, stored procedures, SSIS packages, ETL processes, and reporting transformations into the target architecture.
• Design, build, and optimise Databricks solutions, including Delta tables, Spark SQL transformations, pipelines, quality controls, reconciliation processes, and monitoring capabilities.
• Create source‑to‑target mappings, migration strategies, validation plans, and technical documentation.
• Support business‑as‑usual activities, including SQL development, feed troubleshooting, reporting enhancements, testing, reconciliation, and production issue resolution.

Skills

SQL Server

T‑SQL

Databricks

Spark SQL

Data modelling

Medallion Architecture

ETL migration

Education

Bachelor's degree in CS or related

Tools

SSIS

Job description

This is a key contract role focused on transforming and migrating legacy SQL Server-based data solutions into a modern Databricks architecture.

The successful candidate will assess the current data landscape, identify dependencies and data quality issues, redesign critical data structures, and support the migration of selected workloads into Databricks. You will work closely with business, accounting, and technology stakeholders to ensure the successful delivery of a scalable and supportable data platform.

Key Responsibilities

• Conduct a detailed inventory of the current data environment, including databases, schemas, tables, views, stored procedures, SQL Agent jobs, SSIS packages, feeds, reports, consumers, permissions, and operational processes.
• Document data ownership, lineage, business keys, relationships, calculations, transformations, refresh frequencies, and downstream dependencies.
• Perform data profiling and quality assessments, including completeness, accuracy, duplication, referential integrity, timeliness, and historical coverage.
• Collaborate with business and technical SMEs to determine which data structures, calculations, and reports should be retained, redesigned, consolidated, enhanced, or retired.
• Convert, redesign, and migrate T‑SQL code, stored procedures, SSIS packages, ETL processes, and reporting transformations into the target architecture.
• Design, build, and optimise Databricks solutions, including Delta tables, Spark SQL transformations, pipelines, quality controls, reconciliation processes, and monitoring capabilities.
• Create source‑to‑target mappings, migration strategies, validation plans, and technical documentation.
• Support business‑as‑usual activities, including SQL development, feed troubleshooting, reporting enhancements, testing, reconciliation, and production issue resolution.

Required Skills & Experience

• Extensive experience with SQL Server, T‑SQL, Databricks, and Spark SQL development.
• Strong expertise in data modelling, with a solid understanding of Medallion Architecture.
• Proven track record of migrating SQL Server, SSIS, stored procedures, ETL processes, and reporting workloads to Databricks or similar cloud‑based data platforms.
• Experience with data profiling, metadata management, lineage analysis, dependency mapping, source‑to‑target mappings, and data quality controls.
• Ability to assess complex legacy systems and determine the most effective approach for migration, redesign, consolidation, replacement, or retirement.
• Excellent documentation, stakeholder engagement, communication, and knowledge transfer skills.