Principal Data Engineer: Azure Lakehouse & Databricks

Allianz Commercial; Minnesota; Hybrid; USD 110,000 - 150,000; Full time

Benefits offered by this job

Health insurance

401(K) with company match

Paid holidays

Paid time off

Paid volunteer days

Tuition reimbursement

Paid parental leave

Employee shares program

Employee discounts

Job summary

Allianz Technology seeks a Principal Data Engineer to re-engineer an enterprise data platform on Azure Synapse and migrate workloads to Databricks. You will reverse-engineer pipelines, validate data models, and ensure governance during migration.

Collaborate with governance and architecture teams, mentor engineers, and design lakehouse patterns, SCD/CDC and output validation across parallel runs.

Qualifications

• 7+ years of hands-on experience in Data Engineering, including platform migration or re-engineering work.
• Proven track record working on existing, complex enterprise data platforms - not just building from scratch
• Deep knowledge of enterprise data governance patterns: audit trails, reconciliation, data quality controls, SCD versioning

Responsibilities

• Contribute to the technical assessment and re-engineering of an existing enterprise data platform, spanning all layers from source ingestion through to data consumption
• Reverse-engineer, document, and validate existing pipeline logic, data models, transformation frameworks, and data governance controls
• Identify gaps, defects, and technical debt across the platform and remediate where implementations are incorrect or sub-optimal
• Ensure correctness of data processing patterns including change data capture, slowly changing dimensions, deduplication, and business reconciliation
• Implement target-state designs aligned to modern lakehouse principles, ensuring feature parity and business logic fidelity during transitions
• Support platform evolution initiatives, including parallel-run phases where multiple implementations operate simultaneously, validating output consistency before cutover
• Execute migration of existing workloads to modern data platforms, preserving existing governance and control framework semantics
• Re-implement ingestion, transformation, and orchestration pipelines on target platforms, maintaining audit, quality, and reconciliation standards
• Collaborate with business, data governance, and architecture stakeholders to validate embedded business rules and data quality requirements
• Mentor less experienced engineers, review code and designs, and support decommission planning for legacy components

Skills

Python

SQL

Spark

Data Modeling

ETL/ELT

Tools

Azure DevOps

GitHub Actions

Terraform

ARM

Docker

Job description

Allianz Technology seeks a Principal Data Engineer to re-engineer an enterprise data platform on Azure Synapse and migrate workloads to Databricks. You will reverse-engineer pipelines, validate data models, and ensure governance during migration.

Collaborate with governance and architecture teams, mentor engineers, and design lakehouse patterns, SCD/CDC and output validation across parallel runs.