Senior AI/ML Data Engineer — AI Pipelines & Data Quality

Allstate; United States; Remote; USD 120,000 - 194,000; Full time

Benefits offered by this job

Laptop provided

Monitors provided

Headset provided

Connectivity reimbursement

Job summary

National General, an Allstate company, is seeking an experienced AI/data engineer to design and optimize AI-powered data pipelines that handle structured and unstructured data from multiple sources. You will build LLM-based mapping solutions and ensure accurate, scalable ingestion across enterprise systems.

You will collaborate with product, engineering, and business teams to deliver robust data workflows, automate validation, and drive data quality initiatives in a fast-evolving environment.

Qualifications

• 3+ years of related experience.
• Strong proficiency in Python and SQL.
• Experience with Generative AI / LLMs.
• Experience with data transformation and pipeline development.
• Knowledge of DevOps practices (CI/CD pipelines, version control, cloud deployment).
• Familiarity with Azure.

Responsibilities

• Design and build AI-powered data pipelines to process and transform structured and unstructured data (PDF, Excel, CSV, etc.).
• Develop and optimize LLM-based solutions (prompt engineering, mapping, extraction, and transformation).
• Translate complex data and technical outputs into actionable insights for business stakeholders.
• Partner with cross-functional teams (product, engineering, business SMEs) to define and deliver solutions.
• Validate and improve solution accuracy through testing, iteration, and stakeholder feedback.
• Support integration of ML/AI solutions into enterprise systems and workflows (APIs, databases, cloud platforms).
• Document solutions to ensure scalability, maintainability, and long-term adoption.

Skills

Python

SQL

LLMs

AI/ML

Data pipelines

Azure

CI/CD

Tools

Azure DevOps

C#/.NET

SQL Server

Job description

National General, an Allstate company, is seeking an experienced AI/data engineer to design and optimize AI-powered data pipelines that handle structured and unstructured data from multiple sources. You will build LLM-based mapping solutions and ensure accurate, scalable ingestion across enterprise systems.

You will collaborate with product, engineering, and business teams to deliver robust data workflows, automate validation, and drive data quality initiatives in a fast-evolving environment.