Senior Data Engineer — Spark, Lakehouse, Remote Eligible

Allstate Insurance; Illinois; On-site; USD 90,700 - 153,925; Full time

Benefits offered by this job

Technology setup provided

Connectivity reimbursement for remote/

Job summary

Allstate's Data & Analytics Technology organization is seeking a Data Engineer to design, build, and operate scalable data pipelines that support analytics, reporting, and ML workloads. You will focus on batch and streaming data solutions using Apache Spark and modern cloud data platforms.

You will work with analytics engineers, data scientists, product teams, and platform partners to transform raw data into trusted datasets, ensuring data quality, performance, and governance across the data

Qualifications

• 4+ years of hands-on data engineering experience.
• Experience building production data pipelines with Spark.
• Proficient in Python and SQL.
• Familiar with cloud-based data platforms and lakehouse architectures.
• Knowledge of Microsoft Fabric and OneLake is a plus.

Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines.
• Ingest, transform, and curate data from diverse sources.
• Implement data modeling for reporting and ML workloads.
• Develop lakehouse workloads with Fabric/OneLake.
• Ensure data quality, monitoring, and reconciliation.
• Optimize Spark jobs for performance and cost.
• Collaborate with analytics, BI, and data science teams.

Skills

Apache Spark

Python

SQL

Cloud data platforms

Microsoft Fabric / OneLake

Data modeling

CI/CD / IaC

Distributed data processing

Data quality

Communication skills

Job description

Allstate's Data & Analytics Technology organization is seeking a Data Engineer to design, build, and operate scalable data pipelines that support analytics, reporting, and ML workloads. You will focus on batch and streaming data solutions using Apache Spark and modern cloud data platforms.

You will work with analytics engineers, data scientists, product teams, and platform partners to transform raw data into trusted datasets, ensuring data quality, performance, and governance across the data