Lead Data Engineer (m/f/d)

Allianz Technology; Barcelona; Híbrido; EUR 45.000 - 65.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Up to 25 days working from abroad

Company bonus scheme

Pension and employee shares program

Healthcare and parental leave benefits

Descripción de la vacante

Allianz Technology in Barcelona is seeking a Data Engineer to enhance data management through optimized engineering standards and end-to-end pipeline design.

This role involves collaborating with cross-functional teams to ensure quality and governance in data processing. Ideal candidates will have strong experience with tools like Apache Spark and Azure services, along with a solid understanding of data quality frameworks.

Join us to contribute to innovative solutions in a hybrid work environment!

Formación

• Hands-on expertise building and operating large-scale data pipelines.
• Deep working knowledge of Azure and its services.
• Experience implementing data governance and quality rules.

Responsabilidades

• Define data engineering standards and architecture decisions.
• Design end-to-end data pipelines using tools like Databricks.
• Ensure data quality and governance across pipelines.

Conocimientos

Apache Spark

Kafka

Airflow

Azure Databricks

SQL

Python

Data Quality

CI/CD

Herramientas

Databricks

Azure Synapse

Informatica MDM

Descripción del empleo

About the Job

The SimpliFi Data Pool is a central data repository hosted on Azure, designed to unify OE financial data and enhance cost efficiency. It serves as the primary source of accounting and reporting data, ensuring compliance with mandatory reporting requirements. The Data Pool is a key component of the SimpliFi program, integrated with SAP, providing a harmonized data model with Common Core components, including a globally harmonized Chart of Accounts and standardized accounting and reporting processes.

The ecosystem optimizes finance data management through key components: structured data storage (Staging Area, Raw Data Vault, Business Data Vault), customized feeder system integrations, seamless SAP connectivity via Azure functions, and a PowerBI-enabled reporting layer for in-depth analysis and data lineage transparency. It features Master and Reference Data Management (Informatica IDMC), rigorous data quality checks, and the SimpliFi Studio interface for user-friendly manual uploads and pipeline monitoring, ensuring data accuracy, reliability, and operational efficiency.

What You Do

• Define and enforce data engineering standards — pipeline design patterns, naming conventions, modelling approaches (e.g. Data Vault). Make architecture decisions and translate business/domain requirements into technical solutions.
• Design and oversee end-to-end data pipelines: ingestion, transformation, loading. In SimpliFi terms, this means owning flows across Raw Data Vault, Business Data Vault, UJT, and CORE layers — often spanning tools like Databricks, Azure Synapse, and IDMC.
• Accountable for data quality at the pipeline level — defining checks, monitoring and resolving data issues before they propagate downstream.
• Guide junior/mid engineers, set the bar for code quality, and conduct reviews. A key multiplier for team capability.
• Work closely with Data Modellers, MDM/RDM specialists, Data Architects, and IT component leads. Bridge the gap between technical implementation and business intent.
• Ensure pipelines meet data governance requirements (lineage, classification, access controls) and support gate processes where relevant.
• Act as a technical proxy in planning sessions, helping size effort, flag dependencies and unblock the team during sprint/delivery cycles.

What You Bring

• Hands‑on expertise building and operating large‑scale pipelines — batch and streaming. Proficiency in tools like Apache Spark, Kafka, Airflow and cloud‑native equivalents (ADF, Glue, Dataflow). Experience with ETL/ELT patterns at enterprise scale. Practical experience with modern lakehouse/warehouse platforms — Databricks, Azure Synapse, or equivalents. Understands partitioning, clustering, query optimisation.
• Deep working knowledge of at least one major cloud Azure — compute, storage, networking, managed services. In enterprise contexts like SimpliFi, Azure is typically dominant (ADLS, ADF, Azure Databricks).
• Solid grasp of modelling paradigms — relational, dimensional (Kimball), and Data Vault 2.0. Ability to review and contribute to models, not just implement them. Strong SQL (complex transformations, performance tuning), Python (pipeline logic, data quality scripts), and familiarity with Scala or PySpark for distributed workloads.
• Experience implementing DQ rules, profiling, and monitoring — ideally with tools like Acceldata, Great Expectations, or Monte Carlo.
• Understanding of master and reference data flows — how golden records are created, maintained, and consumed. Experience with platforms like SAP MDG or Informatica MDM is a strong plus.
• CI/CD for data pipelines (GitHub Actions, Azure DevOps), version control discipline, infrastructure‑as‑code basics (Terraform), and containerisation (Docker/Kubernetes awareness).
• Able to read and contribute to architecture documents — understands layers, zones and data flow patterns across a platform. Can engage meaningfully with architects without needing hand‑holding. Know how to implement metadata capture, data lineage (e.g. via IDMC or Purview), and access controls. Experience navigating governance frameworks and gate processes.
• Familiarity with Machine learning is a strong advantage.

What We Offer

• Hybrid work model enabling up to 25 days per year working from abroad.
• Compensation and benefits package that includes a company bonus scheme, pension, employee shares program and various discounts (details vary by location).
• Career development and digital learning programs, international career mobility, and an environment fostering innovation, delivery and empowerment.
• Flexible working, health and wellbeing offers (including healthcare and parental leave benefits) to support family and career balance and help employees return from career breaks.

Equal Opportunity Employer

Allianz Group is one of the most trusted insurance and asset management companies in the world. We are proud to be an equal opportunity employer and encourage you to bring your whole self to work, no matter where you are from, what you look like, who you love, or what you believe in. We welcome applications regardless of race, ethnicity or cultural background, age, gender, nationality, religion, social class, disability, sexual orientation, or any other characteristics protected under applicable local laws and regulations.