Senior Data Engineer Con II

Allstate Insurance Company; Northern (KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Allstate's Data & Analytics Technology organization is seeking a Senior Data Engineer to lead the design, development, and operation of scalable data platforms powering analytics, reporting, AI, and advanced data products.

You will design batch and real-time pipelines using Spark, SQL, Python, and cloud-native technologies, implement Medallion Lakehouse patterns, and collaborate with analytics engineers, data scientists, and product teams.

Qualifications

• Design scalable batch and real-time data pipelines using Spark, SQL, Python and cloud-native technologies.
• Implement Medallion Lakehouse patterns (Bronze-Silver-Gold) to enable enterprise analytics.
• Establish data governance, quality, observability, and lineage across platforms.

Responsibilities

• Lead the design, development, and operation of data platforms for analytics, reporting, and AI.
• Collaborate with analytics engineers, data scientists, and product teams to deliver data assets.
• Optimize pipelines and workloads for performance, reliability, and cost in a large enterprise.

Skills

Apache Spark

SQL

Python

Cloud-native data technologies

Lakehouse design

Data governance

Collaboration

Tools

Microsoft Fabric

OneLake

Job description

At Allstate, great things happen when our people work together to protect families and their belongings from life’s uncertainties. And for more than 90 years, our innovative drive has kept us a step ahead of our customers’ evolving needs. From advocating for seat belts, air bags and graduated driving laws, to being an industry leader in pricing sophistication, telematics, and, more recently, device and identity protection.

Job Description

Allstate's Data & Analytics Technology organization is seeking a Senior Data Engineer to lead the design, development, and operation of scalable, resilient, and high-performing data platforms that power enterprise analytics, reporting, AI, and advanced data products. This role will be responsible for building and optimizing batch and streaming data solutions using Apache Spark, Microsoft Fabric, and modern cloud-native data technologies.

As a Senior Data Engineer, you will play a critical role in designing and implementing Lakehouse architecture based on Medallion (Bronze, Silver, Gold) principles, ensuring that data is governed, trusted, and readily consumable across the enterprise. You will partner closely with analytics engineers, architects, data scientists, and product teams to transform complex data into business-ready information assets through effective data engineering, modeling, and governance practices.

The ideal candidate possesses deep experience in large-scale data processing, dimensional data modeling, and modern software engineering practices, including leveraging Agentic AI-assisted development tools to accelerate engineering productivity, code quality, testing, and operational excellence.

• Design, build, and maintain scalable batch and real-time data pipelines using Apache Spark, SQL, Python, and cloud-native data technologies.
• Architect and implement Medallion Lakehouse data patterns (Bronze, Silver, Gold) to support enterprise analytics, reporting, operational intelligence, and AI use cases.
• Develop and optimize ETL/ELT pipelines that ingest, transform, standardize, and curate data from diverse enterprise source systems.
• Design and implement robust Dimensional models, including fact and dimension tables, to support reporting, self-service analytics, and business intelligence solutions.
• Build and manage data workloads within Microsoft Fabric, OneLake, and modern Lakehouse platforms.
• Establish data quality, observability, and governance frameworks through automated validation, reconciliation, monitoring, and alerting capabilities.
• Optimize Spark workloads for performance, scalability, reliability, and cost efficiency across large and complex datasets.
• Design solutions that effectively manage schema evolution, data lineage, metadata, and source-system changes.
• Develop reusable frameworks, accelerators, and engineering standards that improve consistency and delivery velocity across teams.
• Leverage Agentic AI-assisted development capabilities to accelerate code generation, testing, documentation, troubleshooting, and engineering productivity while adhering to enterprise governance standards.
• Implement CI