Senior Data Engineer: Real-Time Pipelines (Databricks)

Certain Advantage; Greater London; On-site; GBP 95,000 - 130,000; Full time

Benefits offered by this job

Onsite in Central London (3 days/week)

Job summary

Certain Advantage is recruiting a Senior Data Engineer for a central London role focused on scalable data pipelines and real-time analytics. You will work with Databricks, Delta Live Tables and Spark streaming within an Azure cloud setting, ingesting SAP data via Qlik Replicate and ADF to ensure data quality and governance.

The position emphasizes collaboration with cross-functional teams, performance tuning, and robust data architectures to support decision-making across the organization.

Qualifications

• Strong background in Databricks DLT and Delta Lake concepts.
• Solid understanding of streaming technologies and Spark.
• Programming in Python/SQL/Scala and data architecture.

Responsibilities

• Design and implement scalable data pipelines using Databricks and Delta Live Tables.
• Develop streaming data solutions for real-time analytics and decision-making.
• Ingest data from SAP sources using Qlik Replicate and ADF.
• Perform data modelling and optimize data structures for analytics and reporting.
• Collaborate with cross-functional teams to understand data requirements.
• Monitor and optimize data workflows for performance and reliability.
• Maintain technical documentation and contribute to best practices.

Skills

Databricks

Delta Live Tables

Python

SQL

Scala

Spark Structured Streaming

Azure Cloud

Data Modeling

ETL/ELT

Data Governance

Education

Bachelor's or Master’s in CS/Engineering

Tools

Qlik Replicate

Azure Data Factory

Job description

Certain Advantage is recruiting a Senior Data Engineer for a central London role focused on scalable data pipelines and real-time analytics. You will work with Databricks, Delta Live Tables and Spark streaming within an Azure cloud setting, ingesting SAP data via Qlik Replicate and ADF to ensure data quality and governance.

The position emphasizes collaboration with cross-functional teams, performance tuning, and robust data architectures to support decision-making across the organization.