Senior Data Engineer: Spark, Fabric & DataOps Expert

Jobtailor; Belfast City District; On-site; GBP 60,000 - 90,000; Full time

Job summary

Allstate in the United Kingdom is seeking a data engineer to design, develop, and maintain scalable data pipelines. You will work with Spark, Microsoft Fabric, and Lakehouse architectures to ingest, transform, and curate data from diverse enterprise sources.

You will collaborate with Analytics Engineers, BI teams, data scientists, and platform engineers to deliver trusted datasets and data products. Strong mentoring and cloud-based analytics experience are essential.

Qualifications

• Minimum of 2 years of experience in Data Engineering and Big Data technologies.
• Deep expertise in Apache Spark (PySpark/Spark SQL).
• Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and building Semantic Models.
• Strong SQL and data modeling skills.
• Experience building and optimizing ETL/ELT pipelines.
• Experience with Kafka or similar event-streaming platforms.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using Spark and cloud-native platforms.
• Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse source systems.
• Develop data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, and Data Warehouse capabilities.
• Design data models, storage strategies, and architectures to support analytics and reporting.
• Mentor junior engineers through code reviews, design reviews, and guidance.

Skills

Apache Spark

Data Governance

CI/CD

Cloud Analytics

Kafka

Java

Data Modeling

SQL

Observability

Mentoring

Tools

Microsoft Fabric

OneLake

Lakehouse

Data Warehouse

CI/CD

Git

Job description

Allstate in the United Kingdom is seeking a data engineer to design, develop, and maintain scalable data pipelines. You will work with Spark, Microsoft Fabric, and Lakehouse architectures to ingest, transform, and curate data from diverse enterprise sources.

You will collaborate with Analytics Engineers, BI teams, data scientists, and platform engineers to deliver trusted datasets and data products. Strong mentoring and cloud-based analytics experience are essential.