Senior Data Engineer

Certain Advantage; Greater London; On-site; GBP 95,000 - 130,000; Full time

Benefits offered by this job

Onsite in Central London (3 days/week)

Job summary

Certain Advantage is recruiting a Senior Data Engineer for a central London role focused on scalable data pipelines and real-time analytics. You will work with Databricks, Delta Live Tables and Spark streaming within an Azure cloud setting, ingesting SAP data via Qlik Replicate and ADF to ensure data quality and governance.

The position emphasizes collaboration with cross-functional teams, performance tuning, and robust data architectures to support decision-making across the organization.

Qualifications

• Strong background in Databricks DLT and Delta Lake concepts.
• Solid understanding of streaming technologies and Spark.
• Programming in Python/SQL/Scala and data architecture.

Responsibilities

• Design and implement scalable data pipelines using Databricks and Delta Live Tables.
• Develop streaming data solutions for real-time analytics and decision-making.
• Ingest data from SAP sources using Qlik Replicate and ADF.
• Perform data modelling and optimize data structures for analytics and reporting.
• Collaborate with cross-functional teams to understand data requirements.
• Monitor and optimize data workflows for performance and reliability.
• Maintain technical documentation and contribute to best practices.

Skills

Databricks

Delta Live Tables

Python

SQL

Scala

Spark Structured Streaming

Azure Cloud

Data Modeling

ETL/ELT

Data Governance

Education

Bachelor's or Master’s in CS/Engineering

Tools

Qlik Replicate

Azure Data Factory

Job description

Central London - 3 days onsite per week (no flex on this)

Certain Advantage are recruiting on behalf of our trading client for a Senior Data Engineer with strong analytical capabilities and hands-on experience in building scalable data solutions using a programming languages like Python or Scala as well as experience in Databricks and Delta Live Tables - This experience is essential.

Suitable candidates will have expertise in Databricks, streaming architectures, data ingestion frameworks, and dashboarding, along with experience working with SAP data sources.

You will be responsible for designing and implementing scalable data pipelines and enabling real-time and batch analytics across the organization.

Your Responsibilities:

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT).
• Develop and manage streaming data solutions for real-time analytics and decision-making.
• Ingest data from various sources—including SAP systems using Qlik Replicate and ADF to ingest data from various sources ensuring data quality and consistency.
• Perform data modelling and optimize data structures for analytics and reporting.
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions.
• Monitor, troubleshoot, and enhance data workflows for performance and reliability.
• Maintain technical documentation and contribute to data engineering best practices.

Required Skills & Qualifications:

• Strong experience with Databricks- DLT, Delta Lake concepts, UC governance.
• Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader).
• Programming skills in Python, SQL, or Scala.
• Proficiency in data modelling, ETL/ELT processes, and data architecture.
• Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem-solving skills.
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing.
• Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication.
• Experience working in Azure cloud environments.
• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.