Senior Data Engineer — Lakehouse & Spark Platform Lead

Allstate Insurance Company; Northern (KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Allstate's Data & Analytics Technology organization is seeking a Senior Data Engineer to lead the design, development, and operation of scalable data platforms powering analytics, reporting, AI, and advanced data products.

You will design batch and real-time pipelines using Spark, SQL, Python, and cloud-native technologies, implement Medallion Lakehouse patterns, and collaborate with analytics engineers, data scientists, and product teams.

Qualifications

• Design scalable batch and real-time data pipelines using Spark, SQL, Python and cloud-native technologies.
• Implement Medallion Lakehouse patterns (Bronze-Silver-Gold) to enable enterprise analytics.
• Establish data governance, quality, observability, and lineage across platforms.

Responsibilities

• Lead the design, development, and operation of data platforms for analytics, reporting, and AI.
• Collaborate with analytics engineers, data scientists, and product teams to deliver data assets.
• Optimize pipelines and workloads for performance, reliability, and cost in a large enterprise.

Skills

Apache Spark

SQL

Python

Cloud-native data technologies

Lakehouse design

Data governance

Collaboration

Tools

Microsoft Fabric

OneLake

Job description

Allstate's Data & Analytics Technology organization is seeking a Senior Data Engineer to lead the design, development, and operation of scalable data platforms powering analytics, reporting, AI, and advanced data products.

You will design batch and real-time pipelines using Spark, SQL, Python, and cloud-native technologies, implement Medallion Lakehouse patterns, and collaborate with analytics engineers, data scientists, and product teams.