Senior Data Engineer

Norton Blake Limited; Greater London; Ibrido; GBP 111.000 - 122.000; Tempo pieno

Descrizione del lavoro

Norton Blake Limited in London is seeking a Senior Data Engineer for a 6‑month hybrid contract to design scalable data pipelines and enable real‑time analytics across the program.

You will work with Databricks, Delta Live Tables, Python and SQL, building streaming architectures and data ingestion from SAP/ETRM sources, with a focus on performance and cost optimization.

Competenze

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• Strong Databricks and Delta Lake expertise with data governance.
• Proficient in Python and SQL; knowledge of Spark Structured Streaming.

Mansioni

• Design scalable data pipelines using Databricks and Delta Live Tables.
• Develop streaming data solutions for real-time analytics.
• Ingest data from SAP and ETRM sources using Qlik Replicate and ADF.
• Perform data modelling and optimize structures for analytics.
• Collaborate with cross-functional teams to meet data needs.
• Monitor, troubleshoot, and improve data workflows for reliability.
• Maintain technical documentation and best practices.

Conoscenze

Databricks

Delta Live Tables

Streaming analytics

Python

SQL

Scala

Data modelling

ETL/ELT

Azure

Qlik Replicate

ADF

Performance tuning

Formazione

Bachelor's or Master's in Computer Science or Engineering

Strumenti

Qlik Replicate

ADF

Azure

Descrizione del lavoro

Senior Data Engineer, 6 months, London/Hybrid, £600 - £660/day (INSIDE IR35)

Energy/Trading experience is a MUST.

Job Summary:

We are looking for a highly skilled Senior Data Engineer with strong analytical skill, problem solving capabilities and hands‑on experience in building scalable data solutions. The ideal candidate will have expertise in Python, Databricks: PySpak, streaming architectures, data ingestion frameworks, and Performance/Cost Optimization Technique, along with experience working with SAP, ETRM data sources would be an added advantage. The candidate will be responsible for designing and implementing scalable data pipelines and enabling real‑time, batch analytics and across the program.

Key Responsibilities:

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT).
• Develop and manage streaming data solutions for real‑time analytics and decision‑making.
• Ingest data from various sources-including SAP, ETRM systems using Qlik Replicate and ADF to ingest data from various sources ensuring data quality and consistency.
• Perform data modelling and optimize data structures for analytics and reporting.
• Collaborate with cross‑functional teams to understand data requirements and deliver robust solutions.
• Monitor, troubleshoot, and enhance data workflows for performance and reliability.
• Maintain technical documentation and contribute to data engineering best practices.

Required Skills & Qualifications:

• Strong experience with Databricks- DLT, Delta Lake concepts, UC governance.
• Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader).
• Programming skills in Python, SQL, or Scala.
• Proficiency in data modelling, ETL/ELT processes, and data architecture.
• Experience in MS Copilot, AI Foundry , Databricks (Agentbricks, Genie)
• Strong analytical background with problem‑solving skills.
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing.
• Hands‑on expertise in ADF, and Qlik Replicate for data ingestion and replication.
• Experience working in Azure cloud environments.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.