Senior Data Engineer - Lakehouse & Data Pipelines

The Fidelis Partnership; Greater London; On-site; GBP 90,000 - 140,000; Full time

Job summary

The Fidelis Partnership is seeking a Senior Data Engineer to advance an evolving Analytics Data Platform. You will own end-to-end changes, deliver scalable data pipelines, and collaborate with stakeholders to translate business needs into robust data solutions.

You'll work with Lakehouse architectures, Medallion framework, and modern tooling to ensure secure, high-quality data products that power actuarial and reporting functions across the organization.

Qualifications

• Minimum 5 years of experience as a Data Engineer or related role.
• Proven experience designing and developing data pipelines.
• Strong understanding of data quality principles and best practices.
• Built and maintained data products using the Microsoft Data Stack (Azure Data Factory, Data Lake Storage, Fabric or Synapse Analytics).
• Expert-level proficiency in PySpark/Python.
• Applied experience of DBT, AirFlow, or similar for data transformation and orchestration.
• Deep understanding of data modeling, warehousing concepts, and the Medallion (Bronze/Silver/Gold) architecture.
• Excellent organisation and time-management skills, with the ability to work in a dynamic enterprise environment.
• Strong communication skills, able to influence change.
• Ability to work independently and in a team in a fast-paced environment.

Responsibilities

• Deliver data-oriented change projects under Lead Data Engineer guidance.
• Design, develop, implement, and maintain scalable data pipelines.
• Maintain existing data products and BAU enhancements.
• Collaborate with stakeholders to translate needs into technical solutions.
• Transform data in Lakehouses for analysis and reporting.
• Utilise PySpark to create analysis-ready data products.
• Implement automated data quality checks and monitoring.
• Document code and processes for knowledge transfer and maintainability.
• Collaborate with other tech teams to refine requirements and IT solutions.
• Identify bottlenecks and apply best practices across the data platform.
• Deliver long-term technology effectiveness and simplified solutions.

Skills

PySpark/Python

Data pipeline design

Data quality

Azure Data Stack

DBT & AirFlow

Medallion architecture

Stakeholder influence

Leadership/mentoring

Data modelling

Communication

Education

Bachelor's or Master’s in relevant field

Tools

PySpark

SQL

AirFlow

DBT

Azure Data Factory

Job description

The Fidelis Partnership is seeking a Senior Data Engineer to advance an evolving Analytics Data Platform. You will own end-to-end changes, deliver scalable data pipelines, and collaborate with stakeholders to translate business needs into robust data solutions.

You'll work with Lakehouse architectures, Medallion framework, and modern tooling to ensure secure, high-quality data products that power actuarial and reporting functions across the organization.