Senior Consultant II – Software Engineer

Jobtailor; Belfast City District; On-site; GBP 60,000 - 90,000; Full time

Job summary

Allstate in the United Kingdom is seeking a data engineer to design, develop, and maintain scalable data pipelines. You will work with Spark, Microsoft Fabric, and Lakehouse architectures to ingest, transform, and curate data from diverse enterprise sources.

You will collaborate with Analytics Engineers, BI teams, data scientists, and platform engineers to deliver trusted datasets and data products. Strong mentoring and cloud-based analytics experience are essential.

Qualifications

• Minimum of 2 years of experience in Data Engineering and Big Data technologies.
• Deep expertise in Apache Spark (PySpark/Spark SQL).
• Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and building Semantic Models.
• Strong SQL and data modeling skills.
• Experience building and optimizing ETL/ELT pipelines.
• Experience with Kafka or similar event-streaming platforms.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using Spark and cloud-native platforms.
• Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse source systems.
• Develop data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, and Data Warehouse capabilities.
• Design data models, storage strategies, and architectures to support analytics and reporting.
• Mentor junior engineers through code reviews, design reviews, and guidance.

Skills

Apache Spark

Data Governance

CI/CD

Cloud Analytics

Kafka

Java

Data Modeling

SQL

Observability

Mentoring

Tools

Microsoft Fabric

OneLake

Lakehouse

Data Warehouse

CI/CD

Git

Job description

• Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms
• Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse enterprise source systems
• Develop and implement data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, Data Warehouse, and Semantic Model capabilities
• Design data models, storage strategies, and data processing architectures supporting enterprise analytics and reporting
• Partner with Analytics Engineers, BI teams, Data Scientists, Product Managers, Platform Engineers, Security teams, and business stakeholders
• Deliver trusted, reusable, scalable, and analytics-ready datasets and data products
• Drive engineering best practices including code quality, peer reviews, automated testing, CI/CD, and documentation
• Ensure data quality, governance, lineage, metadata management, security, privacy, and compliance requirements
• Monitor and optimize performance, scalability, reliability, and cost efficiency of distributed data processing workloads
• Develop reusable frameworks, libraries, templates, and engineering patterns
• Implement and support CI/CD pipelines, Infrastructure-as-Code principles, and deployment automation
• Troubleshoot complex data integration, performance, and operational issues
• Create and maintain technical documentation, architecture diagrams, operational runbooks, and implementation guides
• Participate in technical design sessions, architecture reviews, and engineering discussions
• Mentor and guide junior and mid-level data engineers through coaching, code reviews, design reviews, and knowledge sharing
• Evaluate and adopt emerging technologies, engineering practices, and AI-assisted development capabilities
• Support production operations through monitoring, root cause analysis, incident resolution, and continuous improvement
• Contribute to Agile delivery processes including sprint planning, estimation, backlog refinement, and execution
• Promote DataOps practices, automation, observability, and engineering excellence
• Serve as a technical subject matter expert for modern data platforms, Spark processing, Microsoft Fabric, and lakehouse architectures

Requirements

• Legal right to work in the UK for employment at Allstate
• Allstate is not providing sponsorship for this vacancy
• Minimum of 2 years of experience in Data Engineering and Big Data technologies
• Deep expertise in Apache Spark (PySpark/Spark SQL)
• Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and building Semantic Models
• Strong SQL and data modeling skills
• Experience building and optimizing ETL/ELT pipelines
• Strong proficiency in Java and experience building distributed applications and services
• Experience with Kafka or similar event-streaming platforms
• Understanding of Data Governance, Data Quality, Security, and Privacy controls
• Ability to mentor junior engineers and provide technical leadership without direct people management responsibilities
• Knowledge of streaming and batch data processing patterns
• Experience with observability solutions including logging, monitoring, tracing, and performance analysis
• Experience working in cloud-based analytics and data platform environments
• Experience with CI/CD, Git-based development, testing, and deployment automation
• Experience working in Agile delivery environments

Core Competencies

Demonstrates expertise in designing and developing scalable data pipelines using Apache Spark and Microsoft Fabric, with a strong focus on ETL/ELT processes, data governance, and cloud-based analytics. Proven ability to mentor junior engineers and drive engineering best practices in Agile environments.

Highest-signal resume keywords

• Apache Spark (PySpark/Spark SQL)
• Microsoft Fabric
• ETL/ELT Pipeline Optimization
• Data Governance and Quality
• CI/CD and Deployment Automation

Hard Skills

• Data Engineering
• Big Data Technologies
• SQL
• Data Modeling
• Java
• Kafka
• Streaming Data Processing
• Observability Solutions
• Cloud-Based Analytics
• Agile Delivery

Soft Skills

• Mentoring
• Technical Leadership
• Collaboration

Industry Keywords

• Data Governance
• Data Quality
• Security
• Privacy Controls
• DataOps

Tools & Technologies

• Apache Spark
• Microsoft Fabric
• OneLake
• Lakehouse Architecture
• Data Warehouse
• CI/CD
• Git