Senior Data Engineer - Spark, Lakehouse, Remote

allstate; Illinois; On-site; USD 90,700 - 153,925; Full time

Benefits offered by this job

Technology setup provided

Connectivity reimbursement for remote/

Remote work eligible with connectivity

Job summary

Allstate is seeking a Data Engineer to design, build, and operate scalable data pipelines supporting enterprise analytics and ML workloads. You will work with Spark, cloud data platforms, and Fabric/OneLake to transform raw data into trusted datasets.

You will partner with analytics engineers, data scientists, and product teams to ensure data quality, performance, and scalable solutions across the data platform.

Qualifications

• Strong experience as a Data Engineer building and operating production data pipelines.
• Hands-on Apache Spark experience for large-scale processing.
• Proficiency in Python, SQL, and data transformation best practices.
• Experience with cloud-based data platforms and storage.
• Familiarity with Microsoft Fabric, OneLake, or similar analytics platforms.
• Experience designing and optimizing data models for analytical workloads.
• Understanding of distributed data processing concepts and fault tolerance.
• Experience with CI/CD, version control, and infrastructure-as-code concepts.
• Excellent communication and collaboration across technical and non-technical teams.
• 4+ years of experience in data engineering or equivalent role.

Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloud-native data technologies.
• Develop and optimize ETL/ELT workflows to ingest and curate data into analytics-ready datasets.
• Implement data modeling and transformation logic to support reporting and ML workloads.
• Build data processing workloads within lakehouse platforms, including Microsoft Fabric / OneLake.
• Ensure data quality, reliability, and consistency with monitoring and validation checks.
• Optimize Spark jobs for performance, cost efficiency, and scalability.
• Manage and evolve data schemas while handling drift and upstream changes.
• Develop reusable frameworks and standardized patterns to improve productivity.
• Implement CI/CD pipelines for data workloads and enable automated testing and deployment.
• Monitor pipelines, troubleshoot failures, and resolve data quality issues.
• Collaborate with analytics engineers, BI developers, and data scientists to deliver curated datasets.
• Work with platform, security, and governance teams to enforce data access controls and compliance.

Skills

Data Engineer

Apache Spark

Python

SQL

Cloud data platforms

Microsoft Fabric

OneLake

Data modeling

CI/CD

Version control

Infrastructure as code

Data quality concepts

Problem solving

Communication skills

Job description

Allstate is seeking a Data Engineer to design, build, and operate scalable data pipelines supporting enterprise analytics and ML workloads. You will work with Spark, cloud data platforms, and Fabric/OneLake to transform raw data into trusted datasets.

You will partner with analytics engineers, data scientists, and product teams to ensure data quality, performance, and scalable solutions across the data platform.