Senior Data Engineer : Databricks & Delta

Eaglecliff Recruitment; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Hybrid work pattern

London Waterloo hybrid model

Job summary

Energy Trading Company is seeking a Senior Data Engineer to design scalable data pipelines using Databricks and Delta Lake, enabling real-time and batch analytics across the organization.

You will work in a hybrid model (London Waterloo) with 3 days in the office and 2 days remote, collaborating with cross-functional teams and handling data ingestion from SAP sources.

Qualifications

• Experience with Databricks and Delta Lake concepts.
• Strong background in streaming ETL/ELT and analytics.
• Experience with SAP data sources and data ingestion pipelines.

Responsibilities

• Design scalable data pipelines with Databricks and Delta Live Tables.
• Develop streaming data solutions for real-time analytics.
• Ingest data from SAP sources using Qlik Replicate and ADF.
• Model data and optimize structures for analytics and reporting.
• Collaborate with cross-functional teams to meet data needs.
• Monitor workflows, troubleshoot performance, and document best practices.

Skills

Databricks & DLT

Streaming architectures

Python/SQL/Scala

Data modelling

Azure cloud

Qlik Replicate

ADF

Problem solving

Tools

Databricks

Qlik Replicate

Azure Data Factory

Delta Lake

Job description

This is an exciting role for a Senior Data Engineer with strong Databricks and Delta to join a world-leading Energy Trading company.

Hybrid: London, Waterloo + Remote - 3 Days in the Office + 2 Days Remote

Role Description:

We are looking for a highly skilled Senior Data Engineer with strong analytical capabilities and hands-on experience in building scalable data solutions.

The ideal candidate will have expertise in Databricks, streaming architectures, data ingestion frameworks, and dashboarding, along with experience working with SAP data sources.

You will be responsible for designing and implementing scalable data pipelines and enabling real-time and batch analytics across the organization.

Key Responsibilities:

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)
• Develop and manage streaming data solutions for real-time analytics and decision-making
• Ingest data from various sourcesincluding SAP systems using Qlik Replicate and ADF to ingest data from various sources ensuring data quality and consistency
• Perform data modelling and optimize data structures for analytics and reporting
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
• Monitor, troubleshoot, and enhance data workflows for performance and reliabilityMaintain technical documentation and contribute to data engineering best practices

Required Skills & Qualifications:

• Strong experience with Databricks- DLT, Delta Lake concepts, UC governance
• Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
• Programming skills in Python, SQL, or Scala
• Proficiency in data modelling, ETL/ELT processes, and data architecture
• Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem-solving skills
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
• Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication
• Experience working in Azure cloud environments