Senior Data Engineer - Spark, Lakehouse & Cloud Pipelines

Allstate Northern Ireland Limited; Belfast City District; Hybrid; GBP 70,000 - 110,000; Full time

Benefits offered by this job

Pension scheme

Corporate bonus scheme

Private medical and dental insurance

Hybrid working

Two paid volunteering days each year

Cycle to work scheme

Flexible working options

Annual performance reviews

Job summary

Allstate Northern Ireland Limited is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines and platforms that enable analytics, AI/ML, and reporting across the organization.

You will lead technical initiatives, mentor engineers, collaborate with Analytics Engineers, Data Scientists, Product Managers, and Platform teams to implement enterprise-scale data solutions in a cloud-first, lakehouse environment.

Qualifications

• 2+ years of experience in Data Engineering and Big Data technologies.
• Deep expertise in Apache Spark (PySpark/Spark SQL).
• Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and building Semantic Models.
• Strong SQL and data modeling skills.
• Experience building and optimizing ETL/ELT pipelines.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms.
• Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse enterprise source systems.
• Develop and implement data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, Data Warehouse, and Semantic Model capabilities.
• Design data models, storage strategies, and data processing architectures that support enterprise analytics and reporting requirements.
• Partner with Analytics Engineers, BI teams, Data Scientists, and business stakeholders to deliver trusted, reusable, and analytics-ready datasets.
• Drive adoption of engineering best practices including code quality, peer reviews, automated testing, CI/CD, and documentation standards.
• Ensure data quality, governance, lineage, metadata management, security, privacy, and compliance requirements are incorporated into data solutions.
• Monitor and optimize performance, scalability, reliability, and cost efficiency of distributed data processing workloads.
• Develop reusable frameworks, libraries, templates, and engineering patterns that improve development productivity and consistency.
• Implement and support CI/CD pipelines, Infrastructure-as-Code principles, and deployment automation for data engineering solutions.
• Troubleshoot complex data integration, performance, and operational issues across platforms and pipelines.
• Collaborate with platform, infrastructure, architecture, and security teams to build resilient, scalable, and secure enterprise data solutions.
• Create and maintain technical documentation, architecture diagrams, operational runbooks, and implementation guides.
• Participate in technical design sessions, architecture reviews, and engineering discussions while providing recommendations and solutions.
• Mentor and guide junior and mid-level data engineers through technical coaching, code reviews, design reviews, and knowledge sharing.
• Evaluate and adopt emerging technologies, engineering practices, and AI-assisted development capabilities to improve delivery velocity and quality.
• Support production operations through monitoring, root cause analysis, incident resolution, and continuous improvement initiatives.
• Contribute to Agile delivery processes including sprint planning, estimation, backlog refinement, and execution of engineering commitments.
• Promote DataOps practices, automation, observability, and engineering excellence across development teams.
• Serve as a technical subject matter expert for modern data platforms, Spark processing, Microsoft Fabric, and lakehouse architectures.

Skills

2+ years Data Engineering

Apache Spark (PySpark/Spark SQL)

Microsoft Fabric / OneLake / Lakehouse

Data Warehouse / Semantic Models

SQL / data modeling

ETL/ELT pipelines

Tools

CI/CD tooling

Job description

Allstate Northern Ireland Limited is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines and platforms that enable analytics, AI/ML, and reporting across the organization.

You will lead technical initiatives, mentor engineers, collaborate with Analytics Engineers, Data Scientists, Product Managers, and Platform teams to implement enterprise-scale data solutions in a cloud-first, lakehouse environment.