Senior Data Engineer - Lakehouse & AI Platform

HGF Limited; Leeds; On-site; GBP 60,000 - 90,000; Full time

Benefits offered by this job

27 days holiday

Bonus scheme

Flexible working

Enhanced parental leave policy

Job summary

HGF Limited is creating a brand-new data engineering role to build and run the technical foundation of its data strategy. You will pull data from core business systems into a Lakehouse on Microsoft Fabric, process it through a medallion architecture, and deliver modeled, trusted data for reporting and AI.

You will work in the Data team reporting to the Head of Data, collaborating with Analysts, IT and Security on secure integration.

Qualifications

• Commercial data engineering experience (4+ years).
• SQL (4+ years).
• Python and PySpark (3+ years).
• Microsoft Fabric or an equivalent Azure data platform (2+ years).
• Medallion / layered lakehouse architecture (2+ years).
• Dimensional data modelling for analytics (2+ years).
• Building and maintaining production data pipelines (3+ years).
• Azure DevOps, including source control and CI/CD practices (2+ years).
• MDM and entity resolution tools such as Aptitude or Senzing (2+ years).

Responsibilities

• Build and maintain scalable data pipelines from core systems and third-party sources.
• Design and develop data solutions with Python, PySpark, SQL, notebooks and dataflows across a medallion architecture.
• Create and optimise data models powering Power BI reporting and analytics.
• Implement data quality, validation and reconciliation processes.
• Monitor, troubleshoot and keep the platform fast, reliable and available.
• Configure master data management and entity resolution for cross-system consistency.
• Own MDM approach including golden records, matching, de-duplication and survivorship rules.

Skills

SQL

Python

PySpark

Azure Fabric

Medallion Architecture

Dimensional Modelling

Data Pipelines

Azure DevOps

MDM/Entity Resolution

Tools

Aptitude

Senzing

Azure DevOps

Microsoft Fabric

Job description

HGF Limited is creating a brand-new data engineering role to build and run the technical foundation of its data strategy. You will pull data from core business systems into a Lakehouse on Microsoft Fabric, process it through a medallion architecture, and deliver modeled, trusted data for reporting and AI.

You will work in the Data team reporting to the Head of Data, collaborating with Analysts, IT and Security on secure integration.