Senior Data Engineer: Spark, Lakehouse & Cloud Data

Allstate; Belfast City District; On-site; GBP 70,000 - 110,000; Full time

Job summary

Allstate is seeking a Senior Data Engineer to design, build, and support scalable data platforms that empower analytics, AI/ML, and data-driven decision making. This role emphasizes hands-on development, technical leadership, and collaboration with cross-functional teams.

You will design pipelines, optimize data solutions, and mentor junior engineers, while ensuring data quality, governance, and cost efficiency across distributed workloads.

Qualifications

• Minimum 2 years of data engineering and big data experience.
• Deep expertise in Apache Spark (PySpark/Spark SQL).
• Strong SQL and data modeling skills.
• Experience with data pipelines ETL/ELT and cloud-native data platforms.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms.
• Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse enterprise source systems.
• Develop and implement data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, Data Warehouse, and Semantic Model capabilities.
• Design data models, storage strategies, and data processing architectures that support enterprise analytics and reporting requirements.
• Partner with Analytics Engineers, BI teams, Data Scientists, and business stakeholders to deliver trusted, reusable, and analytics-ready datasets.
• Drive adoption of engineering best practices including code quality, peer reviews, automated testing, CI/CD, and documentation standards.

Skills

Apache Spark

PySpark

Spark SQL

SQL

ETL/ELT pipelines

Data modeling

Mentoring

Cloud platforms

CI/CD

Kafka

Tools

Microsoft Fabric

OneLake

Lakehouse Architecture

Data Warehouse

Semantic Models

Job description

Allstate is seeking a Senior Data Engineer to design, build, and support scalable data platforms that empower analytics, AI/ML, and data-driven decision making. This role emphasizes hands-on development, technical leadership, and collaboration with cross-functional teams.

You will design pipelines, optimize data solutions, and mentor junior engineers, while ensuring data quality, governance, and cost efficiency across distributed workloads.