Senior Data Engineer – Databricks & DLT

Advantage Resourcing UK Ltd; City Of London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Advantage Resourcing UK Ltd is supporting a trading client to hire a Senior Data Engineer focused on Databricks and Delta Live Tables. You will design scalable pipelines, enable real-time and batch analytics, and partner with cross-functional teams to deliver robust data solutions.

Key responsibilities include ingestion from SAP sources via Qlik Replicate and ADF, data modelling, and performance tuning for reliable insights across the organization.

Qualifications

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.
• Strong experience with Databricks and Delta Live Tables, plus governance concepts.
• Fluent in Python/SQL and data engineering best practices.

Responsibilities

• Design scalable data pipelines using Databricks and Delta Live Tables.
• Develop streaming data solutions for real-time analytics.
• Ingest data from SAP systems and ensure data quality and consistency.
• Perform data modelling and optimize structures for analytics.
• Collaborate with cross-functional teams to meet data requirements.
• Monitor and improve data workflows for reliability and performance.

Skills

Databricks

Delta Live Tables

Python

SQL

Spark Streaming

Data modelling

ETL/ELT

Azure

Qlik Replicate

ADF

Education

CS/Engineering degree

Tools

Qlik Replicate

ADF

Azure

Databricks

Job description

Senior Data Engineer – Databricks & DLT (Delta Live Tables)

Certain Advantage are recruiting on behalf of our trading client for a Senior Data Engineerwith strong analytical capabilities and hands‑on experience in building scalable data solutions using a programming language like Python or Scala as well as experience in Databricks and Delta Live Tables - This experience is essential.

Suitable candidates will have expertise inDatabricks,streaming architectures, data ingestion frameworks, anddashboarding, along with experience working withSAP data sources.

You will be responsible for designing and implementing scalable data pipelines and enabling real‑time and batch analytics across the organization.

Your Responsibilities:

• Design and implement scalable data pipelines usingDatabricks and Delta Live Tables (DLT).
• Develop and managestreaming data solutionsfor real‑time analytics and decision‑making.
• Ingest data from various sources—includingSAP systems using Qlik ReplicateandADFto ingest data from various sources ensuring data quality and consistency.
• Performdata modellingand optimize data structures for analytics and reporting.
• Collaborate with cross‑functional teams to understand data requirements and deliver robust solutions.
• Monitor, troubleshoot, and enhance data workflows for performance and reliability.
• Maintain technical documentation and contribute to data engineering best practices.

Required Skills & Qualifications:

• Strong experience withDatabricks- DLT, Delta Lake concepts, UC governance.
• Solid understanding ofstreaming technologies(e.g., Spark Structured Streaming, Autoloader).
• Programming skills inPython,SQL, or
• Proficiency indata modelling, ETL/ELT processes, and data architecture.
• Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem‑solving skills.
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing.
• Hands‑on expertise inADF, and Qlik Replicatefor data ingestion and replication.
• Experience working inAzure cloud environments.
• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.