Senior Data Engineer

Advantage Resourcing UK Ltd; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

Advantage Resourcing UK Ltd in London is recruiting a Senior Data Engineer for a trading client. You will design scalable data pipelines with Databricks and Delta Live Tables, enabling real-time and batch analytics across the organization.

Required skills include Databricks DLT, Spark Streaming, Python/SQL/Scala, data modelling, ETL/ELT, and experience with ADF and Qlik Replicate in Azure. You will collaborate with cross-functional teams and optimize data workflows.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Strong experience with Databricks DLT and Delta Lake concepts.
• Solid understanding of streaming technologies (Spark Structured Streaming, Autoloader).
• Programming skills in Python, SQL, or Scala.
• Proficiency in data modelling, ETL/ELT processes, and data architecture.
• Hands-on experience with ADF and Qlik Replicate for data ingestion/replication.
• Experience in Azure cloud environments.
• Analytical mindset with strong problem-solving skills.

Responsibilities

• Design and implement scalable data pipelines using Databricks and Delta Live Tables.
• Develop and manage streaming data solutions for real-time analytics and decision-making.
• Ingest data from diverse sources including SAP via Qlik Replicate and ADF.
• Perform data modelling and optimise data structures for analytics.
• Collaborate with cross-functional teams to deliver robust data solutions.
• Monitor, troubleshoot, and optimize data workflows for performance and reliability.
• Maintain technical documentation and contribute to best practices.

Skills

Databricks DLT/Delta Lake

Spark Structured Streaming

Python

SQL

Scala

Data modelling

ETL/ELT

Azure cloud

Qlik Replicate

ADF

MosaicAI/AI Foundry

MLflow/Genie/Agentbricks

Performance tuning

Autoloader

Education

Bachelor’s or Master’s degree in Computer Science/Engineering

Tools

Databricks

Delta Live Tables (DLT)

Qlik Replicate

Azure Data Factory (ADF)

Spark/Autoloader

Job description

Contract

Central London - 3 days onsite per week (no flex on this)

Certain Advantage are recruiting on behalf of our trading client for a Senior Data Engineer with strong analytical capabilities and hands‑on experience in building scalable data solutions using a programming languages like Python or Scala.

We’re looking for candidates who can offer experience in the below so please ensure your application includes your experience with:

• ADF, and Qlik Replicate for data ingestion and replication.

You will be responsible for designing and implementing scalable data pipelines and enabling real-time and batch analytics across the organization.

Your Responsibilities:

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT).
• Develop and manage streaming data solutions for real-time analytics and decision‑making.
• Ingest data from various sources—including SAP systems using Qlik Replicate and ADF to ingest data from various sources ensuring data quality and consistency.
• Perform data modelling and optimise data structures for analytics and reporting.
• Collaborate with cross‑functional teams to understand data requirements and deliver robust solutions.
• Monitor, troubleshoot, and enhance data workflows for performance and reliability.
• Maintain technical documentation and contribute to data engineering best practices.

Required Skills & Qualifications:

• Strong experience with Databricks- DLT, Delta Lake concepts, UC governance.
• Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader).
• Programming skills in Python, SQL, or Scala.
• Proficiency in data modelling, ETL/ELT processes, and data architecture.
• Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem‑solving skills.
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing.
• Hands‑on expertise in ADF, and Qlik Replicate for data ingestion and replication.
• Experience working in Azure cloud environments.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.