Senior Data Engineer - Remote Spark & Lakehouse Expert

Allstate Insurance Company; United States; On-site; USD 90,700 - 153,925; Full time

Benefits offered by this job

Laptop and peripherals

Connectivity reimbursement for home办公

Home-office stipend

Job summary

Allstate Insurance Company is seeking a Data Engineer to design, build, and operate scalable data pipelines supporting enterprise analytics. You will work with analytics engineers, data scientists, and product teams to transform data into trusted datasets and enable data-driven decisions.

The role emphasizes data quality, performance, and operational excellence across the data platform, with a focus on batch and streaming processing using Apache Spark and cloud-native technologies.

Qualifications

• 4+ years of experience in data engineering or equivalent role.
• Strong experience building production data pipelines with Apache Spark.
• Proficiency in Python, SQL, and data transformation best practices.
• Experience with cloud-based data platforms and lakehouse architectures.
• Familiarity with Microsoft Fabric / OneLake or similar analytics platforms.

Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloud-native data technologies.
• Develop and optimize ETL/ELT workflows to ingest, transform, and curate data from diverse source systems into analytics-ready datasets.
• Implement data modeling and transformation logic to support reporting, dashboards, and downstream analytical and machine learning workloads.
• Build and manage data processing workloads within modern lakehouse platforms, including Microsoft Fabric / OneLake (preferred).
• Ensure data quality, reliability, and consistency by implementing validation checks, monitoring, and reconciliation processes.
• Monitor data pipelines and jobs, troubleshoot failures, and resolve performance or data quality issues.

Skills

Apache Spark

Python

SQL

Data Engineering

Tools

Microsoft Fabric

OneLake

Job description

Allstate Insurance Company is seeking a Data Engineer to design, build, and operate scalable data pipelines supporting enterprise analytics. You will work with analytics engineers, data scientists, and product teams to transform data into trusted datasets and enable data-driven decisions.

The role emphasizes data quality, performance, and operational excellence across the data platform, with a focus on batch and streaming processing using Apache Spark and cloud-native technologies.