Remote-Ready Senior Data Engineer: Fabric & Pipelines

Allstate; United States; Remoto; USD 80.000 - 137.000; Tempo pieno

Descrizione del lavoro

Allstate seeks a Senior Data Engineer to design, build, and own scalable data pipelines using Microsoft Fabric and modern data engineering practices. You will partner with product managers, software engineers, and data analysts to deliver trusted data for digital products and analytics.

You will manage end-to-end lifecycle of data products, from design through production support, data quality, and performance against key metrics, while applying CI/CD and governance standards.

Competenze

• Strong proficiency in developing scalable data pipelines using PySpark and Python.
• Hands-on experience with modern data engineering frameworks and tools, including Apache Spark, Kafka, Flink, dbt, or equivalent technologies.
• Strong knowledge of SQL, NoSQL databases, data modeling, and data architecture principles.
• Demonstrated ability to optimize SQL queries, data pipelines, and storage for performance, scalability, and cost efficiency.
• Experience designing and developing batch and/or real-time streaming data solutions.
• Hands-on experience with Microsoft Fabric Notebooks for developing, debugging, testing, and operationalizing data engineering workflows.
• Experience with containerization and orchestration technologies, such as Docker and Kubernetes.
• Experience implementing monitoring and observability solutions, with Datadog experience preferred.
• Ability to work effectively in collaborative, iterative, and product-centric engineering environments.
• Strong communication, analytical, troubleshooting, and problem-solving skills

Mansioni

• Design, build, and own scalable data pipelines and data processing systems.
• Build reusable ingestion, transformation, and storage patterns to support product and analytical needs.
• Integrate data from diverse sources including APIs, databases, event streams, and third‑party systems.
• Implement data models and schemas that create unified, consistent views of business operations.
• Apply modern engineering practices, including version control, testing, CI/CD, and automated deployments.
• Ensure data quality, integrity, and reliability through validation, monitoring, and observability tools.
• Collaborate closely with product managers, engineers, and analysts to understand data requirements.
• Participate in iteration planning, ensuring shared understanding of backlog items and technical needs.
• Engage in daily standups, retrospectives, and product ceremonies as an active member of the team.
• Contribute to data governance practices such as metadata management and data lineage.

Conoscenze

PySpark

Python

SQL

NoSQL databases

Data modeling

Data architecture

Streaming data

Docker

Kubernetes

Datadog

Strumenti

Apache Spark

Kafka

Flink

dbt

Docker

Kubernetes

Descrizione del lavoro

Allstate seeks a Senior Data Engineer to design, build, and own scalable data pipelines using Microsoft Fabric and modern data engineering practices. You will partner with product managers, software engineers, and data analysts to deliver trusted data for digital products and analytics.

You will manage end-to-end lifecycle of data products, from design through production support, data quality, and performance against key metrics, while applying CI/CD and governance standards.