Data Engineer Principal_3003

Allianz Technology; Minneapolis (MN); Hybrid; USD 110,000 - 150,000; Full time

Benefits offered by this job

Health insurance

401(K) with company match

Company paid holidays

Paid time off

Tuition reimbursement

Paid parental leave

Employee shares program

Employee discounts

Job summary

Allianz Technology seeks a senior data engineer to re-engineer a large enterprise data platform, spanning ingestion to consumption. You will reverse-engineer pipelines, validate data models, governance controls, and implement target-state lakehouse designs aligned to modern patterns.

With 7+ years in data engineering, you will migrate workloads to Databricks/Synapse Lakehouse, ensure data quality and governance, and mentor junior engineers while collaborating with governance and business

Qualifications

• Hands-on experience with modern lakehouse architectures and data governance patterns.
• Proven ability to re-engineer complex enterprise data platforms end-to-end.
• Strong skills in SQL, Python and Spark for large-scale data processing.
• Experience migrating workloads to Databricks and Synapse Lakehouse environments.
• Familiarity with CI/CD for data pipelines and infrastructure as code.
• Knowledge of SCD, CDC, data quality controls, and audit trails.

Responsibilities

• Contribute to technical assessment and re-engineering of an enterprise data platform from ingestion to consumption.
• Reverse-engineer and validate pipeline logic, data models, and governance controls.
• Identify gaps and technical debt; remediate sub-optimal implementations.
• Ensure correctness of data processing patterns including CDC and SCD handling.
• Implement target-state designs aligned to lakehouse principles with feature parity.
• Support evolution initiatives with parallel-run validations before cutover.
• Migrate workloads to modern data platforms while preserving governance semantics.
• Re-implement pipelines on target platforms; maintain audit, quality, and reconciliation standards.
• Collaborate with stakeholders to validate embedded business rules and data quality.
• Mentor engineers, review designs, and support decommissioning of legacy components.

Skills

Azure Synapse

Databricks

PySpark

Python

SQL

Data governance

SCD CDC

Lakehouse design

Data modeling

CI/CD

Terraform

Docker

YAML

Stakeholder collaboration

Tools

ADLS Gen2

Delta Lake

Oracle GoldenGate

Power BI

Azure SQL Data

Unity Catalog

Purview

Job description

Key Responsibilities

• Contribute to the technical assessment and re-engineering of an existing enterprise data platform, spanning all layers from source ingestion through to data consumption
• Reverse-engineer, document, and validate existing pipeline logic, data models, transformation frameworks, and data governance controls
• Identify gaps, defects, and technical debt across the platform and remediate where implementations are incorrect or sub-optimal
• Ensure correctness of data processing patterns including change data capture, slowly changing dimensions, deduplication, and business reconciliation
• Implement target-state designs aligned to modern lakehouse principles, ensuring feature parity and business logic fidelity during transitions
• Support platform evolution initiatives, including parallel-run phases where multiple implementations operate simultaneously, validating output consistency before cutover
• Execute migration of existing workloads to modern data platforms, preserving existing governance and control framework semantics
• Re-implement ingestion, transformation, and orchestration pipelines on target platforms, maintaining audit, quality, and reconciliation standards
• Collaborate with business, data governance, and architecture stakeholders to validate embedded business rules and data quality requirements
• Mentor less experienced engineers, review code and designs, and support decommission planning for legacy components

Core Technical Skills

• Azure Synapse & Data Platform Mandatory hands-on expertise with:
• Azure Synapse Analytics (Pipelines, Spark Pool, Dedicated SQL Pool)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Delta Lake on Azure (Synapse Lakehouse patterns)
• Oracle Golden Gate Replication for real-time source integration
• Azure Analysis Services and Power BI consumption layer patterns
• Deep understanding of medallion architecture: Raw / Harmonized / Conformed / Consumption layers
• Strong knowledge of SCD Type 0/1/2, CDC patterns, soft/hard delete, and retroactive change processing
• Experience with Synapse SQL Pool — stored procedures, control tables, and data quality validation patterns
• Experience with audit, balance, and control frameworks — parameterized, modular pipeline governance at enterprise scale
• Familiarity with config-driven and automation-first pipeline patterns (YAML, PySpark, SQL-driven generation from mapping documents)

Databricks & Lakehouse

• Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred)
• Strong Apache Spark skills (PySpark / Spark SQL)
• Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse
• Ability to re-implement governance and control frameworks natively in Databricks (audit logging, reconciliation, DQ checks)
• Experience with Delta Lake features: MERGE, CDC, time travel, schema enforcement

Data Engineering & Development

• Strong Python and SQL programming skills
• Experience with ETL/ELT at scale: denormalization, surrogate keys, directory tables, curated data models
• Experience integrating complex data sources: Oracle DB, SQL Server, Azure SQL DB, file systems, Salesforce, APIs
• Strong data modelling skills: relational, dimensional, and lakehouse-oriented

DevOps & Automation

• CI/CD pipelines for data engineering (Azure DevOps / GitHub Actions)
• Infrastructure as Code (Terraform or ARM)
• Containerization (Docker)
• Experience with automated testing frameworks for data pipelines (unit testing, reconciliation-based validation)

Nice to Have

• Experience with Unity Catalog for data governance and lineage
• Familiarity with Azure Purview for data cataloguing and governance
• Exposure to real-time and streaming pipelines (Event Hub / Kafka / Kinesis)
• Experience with GenAI or ML platform integration (MLOps, feature engineering pipelines)
• Familiarity with monitoring and observability tools (e.g., Dynatrace)
• Exposure to BI tools (Power BI, Tableau)

Experience & Profile

• 7+ years of hands-on experience in Data Engineering, including platform migration or re-engineering work
• Proven track record working on existing, complex enterprise data platforms — not just building from scratch
• Deep knowledge of enterprise data governance patterns: audit trails, reconciliation, data quality controls, SCD versioning
• Strong analytical mindset: ability to read existing implementations, identify intent versus defect, and make sound re-engineering decisions
• Primarily hands-on, while comfortable contributing to technical design discussions
• Strong communication skills — able to engage business, governance, and engineering stakeholders with clarity
• Experience working in regulated or enterprise-scale environments (financial services a plus)

What We Offer

• We offer a hybrid work model which recognizes the value of striking a balance between in-person collaboration and remote working
• We believe in rewarding performance and our compensation and benefits package includes health insurance, company bonus scheme, 401(K) with company match, company paid holidays, paid time off, paid volunteer days, tuition reimbursement, paid parental leave, an employee shares program and employee discounts From career development and digital learning programs to international career mobility, we offer lifelong learning for our employees worldwide and an environment where innovation, delivery and empowerment are fostered.
• The annualized base pay range for this role is $110,000 - 150,000. The annual base salary range represents a nationwide market range. The actual salary for this position will be determined by a number of factors, including the scope, complexity and location of the role, and the skills, education, training, credentials and experience of the candidate. The base pay is just one component of the ATA total compensation package. As part of our comprehensive compensation and highly rated benefits programs, ATA also offers eligibility for an incentive-based annual bonus.

About Allianz Technology

Allianz Group is one of the most trusted insurance and asset management companies in the world. Caring for our employees, their ambitions, dreams and challenges is what makes us a unique employer.

We are united by a shared commitment: to put our customers first and at the centre of everything we do. Their needs inspire our thinking and guide our actions.

Together, we can build an environment where everyone feels empowered and confident to explore, grow and shape a better future – for our customers and for the world around us. At Allianz, we stand for unity: we believe that a united world is a more prosperous world, and we are dedicated to consistently advocating for equal opportunities for all. The foundation for this is our inclusive workplace, where people and performance both matter, and where integrity, fairness, inclusion and trust are at the heart of our culture.

We therefore welcome applications regardless of ethnicity or cultural background, age, gender, nationality, religion, social class, disability or sexual orientation, or any other characteristics protected under applicable local laws and regulations.

Join us. Let’s care for tomorrow. You.IT