Data Platform Engineer Principal_999

Allianz Technology; Minneapolis (MN); Hybrid; USD 130,000 - 170,000; Full time

Benefits offered by this job

Health insurance

Company bonus scheme

401(K) with company match

Paid holidays

Paid time off

Paid volunteer days

Tuition reimbursement

Paid parental leave

Employee shares program

Employee discounts

Job summary

Allianz Technology in the United States is seeking a seasoned cloud platform architect to design and govern data platform engineering across Databricks, Azure, and Snowflake. You will drive scalable architectures, security compliance, and cost-conscious operations.

Lead reference architectures, mentor engineers, and collaborate with security, data engineering, and enterprise architecture to shape the platform roadmap. Hybrid work is supported with a competitive package.

Qualifications

• 5+ years in cloud infra/platform engineering with data platforms.
• Hands-on Databricks Unity Catalog, Delta Lake, and performance tuning.
• Advanced Azure networking, IAM, ADLS Gen2 and Data Factory/Synapse.
• Snowflake multi-cluster warehouses, data sharing, and Snowpipe/Tasks.
• Infrastructure-as-Code via Terraform; scripting in Python/PowerShell.

Responsibilities

• Design and implement Databricks architecture and job orchestration.
• Architect Azure data landing zones and VNet integration.
• Design Snowflake account-level architecture and data sharing.
• Define reference architectures and govern platform reviews.

Skills

Databricks expertise

Azure engineering

Snowflake engineering

Terraform IaC

Python scripting

Security & compliance

Tools

Databricks

Snowflake

Azure

Terraform

GitHub Actions

OPA/Conftest

Job description

• Design and implement Databricks architecture: workspace topology, Unity Catalog metastore design, cluster policies, compute pools, Delta Lake optimization (Z-ordering, OPTIMIZE/VACUUM scheduling, liquid clustering), and job orchestration strategy (Workflows, multi-task jobs, Delta Live Tables)
• Architect Azure data landing zones: subscription/resource group hierarchy, hub-spoke networking, private endpoints, VNet integration for Databricks (VNet injection/secure cluster connectivity), ADLS Gen2 hierarchical namespace design, and Azure Data Factory/Synapse pipeline orchestration
• Design Snowflake account-level architecture: multi-cluster warehouse strategy, resource monitor hierarchies, query performance tuning, data sharing/clean rooms, Snowpipe/Streams/Tasks for ingestion, and cost attribution models across business units
• Define reference architectures and golden patterns for new data platform use cases; act as technical authority in architecture review boards

What You Do

• Design and implement Databricks architecture: workspace topology, Unity Catalog metastore design, cluster policies, compute pools, Delta Lake optimization (Z-ordering, OPTIMIZE/VACUUM scheduling, liquid clustering), and job orchestration strategy (Workflows, multi-task jobs, Delta Live Tables)
• Architect Azure data landing zones: subscription/resource group hierarchy, hub-spoke networking, private endpoints, VNet integration for Databricks (VNet injection/secure cluster connectivity), ADLS Gen2 hierarchical namespace design, and Azure Data Factory/Synapse pipeline orchestration
• Design Snowflake account-level architecture: multi-cluster warehouse strategy, resource monitor hierarchies, query performance tuning, data sharing/clean rooms, Snowpipe/Streams/Tasks for ingestion, and cost attribution models across business units
• Define reference architectures and golden patterns for new data platform use cases; act as technical authority in architecture review boards

Access Management & Security Engineering

• Design fine-grained authorization models: Unity Catalog row/column-level security and dynamic views, Snowflake RBAC hierarchies with functional/access roles separation, Azure RBAC + PIM for just-in-time privileged access
• Engineer identity federation across platforms: Entra ID SCIM provisioning into Databricks/Snowflake, OAuth/SAML SSO, service principal and managed identity lifecycle management, secrets management (Azure Key Vault, Databricks secret scopes)
• Lead security hardening: private connectivity (Private Link/Service Endpoints), network security group design, customer-managed encryption keys (CMK), audit log pipeline design (Unity Catalog audit logs, Snowflake ACCOUNT_USAGE, Azure Activity Logs) feeding into SIEM
• Own technical response to security findings and support DORA/GDPR compliance evidence generation, including data residency and data lineage requirements

Cost & Performance Engineering

• Build cost observability tooling/dashboards correlating Azure cost management, Databricks DBU consumption, and Snowflake credit usage against business units and workloads
• Engineer compute efficiency: cluster autoscaling policies, spot/low-priority instance strategies, Photon acceleration adoption, warehouse auto-suspend tuning, query profiling and optimization (pruning, caching, clustering keys)
• Implement FinOps automation: policy-as-code guardrails (cluster size limits, idle timeout enforcement), chargeback/showback models, anomaly detection on spend

Reliability Engineering & Operations

• Define and operate SLOs/SLIs for platform availability and job reliability; build observability stacks (metrics, logs, traces) using Azure Monitor, Log Analytics, and Databricks system tables
• Design disaster recovery architecture: cross-region replication for Delta Lake/ADLS, Snowflake database replication and failover groups, RTO/RPO-driven DR runbooks and tested failover procedures
• Build CI/CD pipelines for infrastructure and platform configuration using Terraform (modules for Databricks/Azure/Snowflake providers), GitHub Actions/Azure DevOps, with automated drift detection and policy validation (OPA/Conftest, Checkov)
• Lead incident response for Sev1/2 platform incidents; drive blameless postmortems and reliability improvements

Technical Leadership & Governance

• Mentor junior/mid-level cloud administrators and data engineers on platform best practices
• Define data platform engineering standards, IaC module libraries, and reusable automation frameworks
• Partner with data engineering, security, and enterprise architecture to shape platform roadmap
• Represent the data platform team in architecture governance and change advisory processes

What You Bring

• 5+ years in cloud infrastructure/platform engineering, with 3+ years focused on data platforms
• Deep hands‑on expertise with Databricks: Unity Catalog, Delta Lake internals, cluster/job architecture, performance tuning at scale
• Advanced Microsoft Azure engineering experience: networking (VNets, Private Link, ExpressRoute), IAM/Entra ID, ADLS Gen2, Azure Data Factory/Synapse
• Advanced Snowflake engineering experience: warehouse/query optimization, RBAC design, data sharing, Snowpipe/Streams/Tasks
• Strong Infrastructure-as-Code expertise: Terraform (module authoring, state management, provider-level configuration for Databricks/Azure/Snowflake)
• Strong scripting/automation: Python (API integrations, automation frameworks), PowerShell, Bash
• Solid understanding of distributed systems, query engines (Spark internals), and data lake/lakehouse architecture patterns
• Experience designing for compliance in regulated environments (DORA, GDPR, data residency/lineage requirements)
• Proven incident response and reliability engineering experience (SLO design, postmortems, DR testing)

Preferred Qualifications

• Advanced certifications: Databricks Certified Data Engineer Professional, SnowPro Advanced Architect, Azure Solutions Architect Expert
• Experience in financial services/insurance with hands‑on DORA/GDPR implementation
• Experience with policy-as-code tooling (OPA, Sentinel, Checkov) and GitOps workflows
• Familiarity with data lineage/catalog tools (Unity Catalog, Purview) and observability platforms (Azure Monitor, Datadog)
• Experience building internal developer platforms or self‑service data infrastructure tooling

What We Offer

• We offer a hybrid work model which recognizes the value of striking a balance between in-person collaboration and remote working
• We believe in rewarding performance and our compensation and benefits package includes health insurance, company bonus scheme, 401(K) with company match, company paid holidays, paid time off, paid volunteer days, tuition reimbursement, paid parental leave, an employee shares program and employee discounts From career development and digital learning programs to international career mobility, we offer lifelong learning for our employees worldwide and an environment where innovation, delivery and empowerment are fostered.
• The annualized base pay range for this role is $130,000 - 170,000. The annual base salary range represents a nationwide market range. The actual salary for this position will be determined by a number of factors, including the scope, complexity and location of the role, and the skills, education, training, credentials and experience of the candidate. The base pay is just one component of the ATA total compensation package. As part of our comprehensive compensation and highly rated benefits programs, ATA also offers eligibility for an incentive-based annual bonus.

About Allianz Technology

• Allianz Group is one of the most trusted insurance and asset management companies in the world. Caring for our employees, their ambitions, dreams and challenges is what makes us a unique employer.
• We are united by a shared commitment: to put our customers first and at the centre of everything we do. Their needs inspire our thinking and guide our actions.
• Together, we can build an environment where everyone feels empowered and confident to explore, grow and shape a better future – for our customers and for the world around us. At Allianz, we stand for unity: we believe that a united world is a more prosperous world, and we are dedicated to consistently advocating for equal opportunities for all. The foundation for this is our inclusive workplace, where people and performance both matter, and where integrity, fairness, inclusion and trust are at the heart of our culture.
• We therefore welcome applications regardless of ethnicity or cultural background, age, gender, nationality, religion, social class, disability or sexual orientation, or any other characteristics protected under applicable local laws and regulations.

Let’s care for tomorrow.

You.IT