Senior Data Engineer — Lakehouse Analytics & PySpark

Fidelis Partnership; Greater London; Vor Ort; GBP 80.000 - 120.000; Vollzeit

Zusammenfassung

The Fidelis Partnership seeks a Senior Data Engineer to advance our Analytics Data Platform, delivering scalable data pipelines and end-to-end data solutions. You will translate business needs into robust technical designs, mentor junior engineers, and collaborate with cross-functional teams to implement secure, high-quality data products.

You will work with cutting-edge technologies like Lakehouse architectures, PySpark, and the Microsoft data stack, driving continuous improvement in data

Qualifikationen

• 5+ years of experience as a Data Engineer or related role
• Proven experience designing and developing data pipelines
• Strong understanding of data quality principles and best practices
• Experience with the Microsoft Data Stack (Azure Data Factory, Data Lake Storage, Fabric or Synapse Analytics)
• Expert-level proficiency in PySpark/Python
• Applied experience of DBT, AirFlow, or similar for data transformation and orchestration
• Deep understanding of data modeling, warehousing concepts, and the Medallion architecture
• Excellent organisation and time-management, in a fast-paced environment
• Strong communication skills and ability to influence change
• Experience mentoring junior engineers is a plus

Aufgaben

• Deliver data-oriented change projects under the Lead Data Engineer
• Design, develop, implement, and maintain scalable and secure data pipelines
• Share responsibility for BAU enhancement and maintenance of data products
• Collaborate with stakeholders to translate needs into technical solutions
• Transform data in Lakehouses and deliver clean, analysis-ready data
• Utilise PySpark to produce high-quality data products
• Implement automated data quality checks and monitoring for Data Ops
• Document code and processes for knowledge transfer
• Work with other tech teams to refine requirements and translate needs into IT solutions
• Identify bottlenecks in the data platform and implement best practices
• Deliver long-term technology effectiveness and sensible controls
• Foster a culture of ownership, accountability and technical excellence
• Develop understanding of industry and products/services
• Stay updated on data engineering tech and best practices
• Mentor and support junior engineers and team culture

Kenntnisse

Data pipeline design

PySpark/Python

Azure data stack familiarity

DBT/AirFlow

Medallion architecture

Data quality

Communication skills

Leadership

Mentoring juniors

Machine learning familiarity

Tools

Azure Data Factory

Data Lake Storage

Fabric

Synapse Analytics

DBT

AirFlow

Medallion architecture

Jobbeschreibung

The Fidelis Partnership seeks a Senior Data Engineer to advance our Analytics Data Platform, delivering scalable data pipelines and end-to-end data solutions. You will translate business needs into robust technical designs, mentor junior engineers, and collaborate with cross-functional teams to implement secure, high-quality data products.

You will work with cutting-edge technologies like Lakehouse architectures, PySpark, and the Microsoft data stack, driving continuous improvement in data