Lead Data Engineer - AI-Ready Fabric Platform

Peter Millar; Milwaukee (WI); On-site; USD 150,000 - 210,000; Full time

Job summary

Peter Millar is seeking a Lead/Principal Data Engineer to own the modern data platform built on Microsoft Fabric and OneLake. You will guide architecture, establish CI/CD standards, and partner with outsourced teams to elevate data engineering and AI-ready data foundations for analytics and GenAI.

The role emphasizes end-to-end platform ownership, security controls, and collaboration with AI engineering teams to integrate Azure AI services into production data workflows.

Qualifications

• 8+ years in data engineering / data architecture, incl. 2+ years in lead capacity.
• Expert SQL and Python; strong Spark (PySpark).
• Proven track record building/operating a Microsoft Fabric / OneLake platform in production.
• Experience managing or directing outsourced engineering teams.
• Exposure to Microsoft Purview, data governance, and security/compliance frameworks.
• Bachelor’s in Computer Science, Engineering, or related field; Master’s or relevant Microsoft certifications a plus.

Responsibilities

• Own end-to-end architecture of the Fabric environment, including Lakehouse, Warehouse, medallion pattern.
• Govern OneLake with domains, shortcuts, mirroring, and Delta Lake/Parquet standards.
• Ingest and transform data at scale with Fabric Data Factory, Dataflows Gen2, Spark notebooks.
• Define and enforce engineering standards: Git-based source control, CI/CD, deployment pipelines.

Skills

Microsoft Fabric

OneLake

Azure AI

Microsoft Foundry

Azure data platform

Engineering practice

Education

Bachelor's in Computer Science or related field

Master’s or Microsoft certifications (Fabric Analytics Engineer, Azure Data Engineer Associate) a plus

Job description

Peter Millar is seeking a Lead/Principal Data Engineer to own the modern data platform built on Microsoft Fabric and OneLake. You will guide architecture, establish CI/CD standards, and partner with outsourced teams to elevate data engineering and AI-ready data foundations for analytics and GenAI.

The role emphasizes end-to-end platform ownership, security controls, and collaboration with AI engineering teams to integrate Azure AI services into production data workflows.