Senior Data Engineer - Spark & Lakehouse (Remote)

Allstate Northern Ireland Limited; Illinois; On-site; USD 90,700 - 153,925; Full time

Job summary

Allstate Northern Ireland Limited is seeking a Senior Data Engineer Con II to design, build, and operate scalable data pipelines using Apache Spark and modern cloud data platforms. The role emphasizes data quality, performance, and governance while collaborating with analytics engineers and data scientists.

The candidate should have 4+ years of data engineering experience, strong Python and SQL skills, and familiarity with Microsoft Fabric / OneLake.

Qualifications

• 4+ years of experience in data engineering or equivalent role (preferred).
• Hands-on experience with Apache Spark for large-scale data processing.
• Proficiency in Python, SQL, and data transformation best practices.

Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloud-native data technologies.
• Develop and optimize ETL/ELT workflows to ingest, transform, and curate data for analytics-ready datasets.
• Collaborate with analytics engineers, data scientists, and product teams to deliver curated datasets and ensure data quality.

Skills

Data engineering

Python

SQL

Big data concepts

Cloud data platforms

Tools

Apache Spark

Microsoft Fabric

OneLake

Job description

Allstate Northern Ireland Limited is seeking a Senior Data Engineer Con II to design, build, and operate scalable data pipelines using Apache Spark and modern cloud data platforms. The role emphasizes data quality, performance, and governance while collaborating with analytics engineers and data scientists.

The candidate should have 4+ years of data engineering experience, strong Python and SQL skills, and familiarity with Microsoft Fabric / OneLake.