Data Engineer for AI (f/m/d)

Allianz Global Investors; Frankfurt; Hybrid; EUR 90,000 - 120,000; Full time

Benefits offered by this job

Hybrid work model

Company pension plan

Relocation assistance

Job summary

Allianz Global Investors seeks a Data Engineer for AI to design, build and operate data products on a Databricks and Foundry-based platform. You will enable self-service data access for AI/ML use cases while upholding security, governance and compliance-by-design principles.

The role emphasizes building reusable pipelines, feature stores, and GenAI data workflows, with L3 support and collaboration across Data Scientists, AI Engineers and DevOps. Frankfurt-based, with hybrid work options.

Qualifications

• Minimum 3 years building and operating data pipelines at scale on Databricks (batch/streaming).
• Strong understanding of AI/ML data engineering concepts and data quality.
• Experience with governance, security and regulatory compliance in data platforms.
• Proficient in CI/CD and infrastructure-as-code (Terraform) on Azure.

Responsibilities

• Develop, maintain and enhance standardized data pipelines and reusable components for AI/ML data prep.
• Build and operate batch and streaming data pipelines on Databricks for trusted data products.
• Create Gold datasets and feature stores with versioning and documentation.
• Collaborate with ML/AI engineers on GenAI data prep and RAG pipelines.
• Ensure security, lineage, retention and auditing via Unity Catalog.

Skills

Databricks

Spark

Python

SQL

CI/CD

Terraform

Azure

Education

Diploma in Computer Science

Tools

Microsoft Foundry

Job description

In the role of Data Engineer for AI (f/m/d), you will design, build and operate data products and reusable data preparation components on the Data and AI Platform at Allianz Global Investors, built on Databricks and Foundry.

Your focus is to enable platform customers to reliably source, ingest, transform, validate and serve high-quality, compliant data for AI and ML use cases (including analytics and GenAI), so teams can consume data in a self-service manner. You will provide technical guidance and L3 support to delivery teams, define and promote best practices for data pipelines and data quality, and ensure adherence to security, privacy and regulatory requirements following a compliance-by-design approach.

In addition, you will continuously evolve the platform’s data engineering capabilities by integrating new features (e.g., ingestion patterns, transformation frameworks, governance controls and monitoring) and by delivering standardized, reusable pipelines and templates that scale across use cases. Databricks is the platform’s primary technology for data engineering, with Microsoft Foundry part of its wider scope.

This position will be based in Frankfurt.

What You Will Do

• Reusable data preparation building blocks & ML-ready pipelines: Develop, maintain and enhance standardized pipeline patterns, templates and utilities (ingestion, transformation, validation, enrichment) to deliver AI/ML-ready datasets. Build robust pipelines for structured and unstructured data with reproducible, deterministic outputs
• Data pipeline engineering on Databricks: Build and operate reliable batch and streaming pipelines to ingest data from internal and external sources, curate datasets, and publish trusted data products for AI, ML and analytical consumption
• Curated data products (Bronze/Silver/Gold): Establish controlled dataset evolution, enforce data quality and freshness, and curate gold tables explicitly designed for analytical, ML and AI use cases.
• Feature engineering & feature store: Design, build and operate feature engineering pipelines and curate a ready-to-use feature store with proper versioning and lightweight documentation (ownership, purpose, inputs/outputs, SLAs)
• GenAI / RAG data & embedding pipelines: Collaborate with ML and AI Engineers to build and operate pipelines for GenAI/RAG systems, including new data ingestion, Bronze/Silver updates, chunking, embedding refresh and index updates to maintain knowledge base relevance
• Systematic experiments & model improvement loops: Enable systematic experimentation (offline evaluations, A/B tests) and support model retraining using user feedback (binary and non-binary) to continuously improve models
• Platform capability evolution (Data for AI): Collaborate with the Data and AI Platform Architect to introduce and operationalize platform capabilities for ingestion, transformation and serving (Delta/Unity Catalog patterns, orchestration, monitoring), making them available securely and consistently
• New data source onboarding & Lakeflow connectors: Facilitate provisioning and rollout of Databricks Lakeflow connectors to integrate new data sources and standardize ingestion across domains
• Security, governance & compliance-by-design: Implement and enforce governance using Unity Catalog, including access control, lineage, retention, encryption and auditing, ensuring compliance with internal standards and regulations
• Data quality, validation & reliability: Establish automated data quality checks and validations (e.g., DQX), define SLAs/SLOs for freshness and reliability, and implement monitoring and observability
• Performance & cost optimisation: Optimize Spark workloads, storage layouts and compute usage to ensure scalable, stable and cost-efficient data pipelines
• Collaboration & enablement: Partner with Data Scientists, ML Engineers, AI CoE, DevOps and SecOps to define data requirements, promote reusable patterns, and enable self-service through documentation, coaching and reviews
• Operations & support: Provide L3 support for pipeline and data product incidents, perform root‑cause analysis, and drive continuous improvements to platform stability and customer satisfaction
• Integration & data product exposure: Integrate Databricks with (Azure) cloud services, enterprise systems, external providers and SaaS platforms, and expose curated data products securely to downstream consumers (feature stores, model training/inference, BI)

Required

What you bring

• Minimum 3 years of practical experience building and operating data pipelines and data products at scale on Databricks (batch and/or streaming), ideally in environments with strong governance requirements
• Strong understanding of data engineering concepts for AI/ML (data modelling, data quality, data contracts, feature engineering collaboration, reproducibility) and how data characteristics impact model outcomes
• Expert‑level technology skills in Databricks, Spark, Python and SQL, plus solid engineering practices such as CI/CD and infrastructure‑as‑code (e.g., Terraform) on Azure
• Diploma in computer science or a similar field
• Very good understanding of data architecture and platform patterns (Lakehouse concepts, Delta, medallion approaches, data product thinking) and how to operationalize them in a governed enterprise context
• Excellent analytical, planning, and organizational skills
• Good oral and written communication skills in English
• Ability to efficiently and effectively document and share knowledge and enable others

Preferred

• Working experience in the financial industry, preferably in asset management
• International work experience
• Certification “Databricks Certified Data Engineer Professional” (or equivalent)
• Certification “Databricks Certified Data Engineer Associate” (or equivalent)
• Optional: Certification “Databricks Certified Generative AI Engineer Associate” (helpful for GenAI‑related data preparation patterns)
• Hands‑on exposure to Microsoft Foundry (Azure AI Foundry), or a strong willingness to build expertise in it

What We Offer

• We empower our employees by ensuring flexible work arrangements that maintain a balance between performance, productivity, career development and personal priorities (e.g., hybrid model/ flexible working hours)
• Securing your future: Access to company pension/savings plans
• Family support (relocation/ childcare facilities)
• Company share purchasing plan
• Mental health and wellbeing programs
• Mobility solutions (Jobrad bike leasing, subvention Jobticket)
• Career opportunities within the entire Allianz Group
• Self‑guided learning & development
• Volunteering time
• … and so much more!

About Allianz Global Investors

Allianz Global Investors is a leading active asset manager, managing assets for individuals, families,and institutions worldwide. Our shared goal to shape a future of opportunity for our clients is foundational to our culture and our values. We look to attract, and actively nurture, individuals with the drive to achieve excellence at every stage of the investment experience.

We believe an inclusive culture is key to a flourishing workforce. Not only do we celebrate the broad range of experiences, backgrounds, and attributes that our workforce brings, but we support our people to live and work in a way that promotes their own wellbeing.

Allianz Group is one of the most trusted insurance and asset management companies in the world. Caring for our employees, their ambitions, dreams and challenges is what makes us a unique employer.

We are united by a shared commitment: to put our clients first and at the centre of everything we do. Their needs inspire our thinking and guide our actions.

Together, we can build an environment where everyone feels empowered and confident to explore, grow and shape a better future – for our clients and for the world around us. At Allianz, we stand for unity: we believe that a united world is a more prosperous world, and we are dedicated to consistently advocating for equal opportunities for all. The foundation for this is our inclusive workplace, where people and performance both matter, and where integrity, fairness, inclusion and trust are at the heart of our culture.

We therefore welcome applications regardless of ethnicity or cultural background, age, gender, nationality, religion, social class, disability or sexual orientation, or any other characteristics protected under applicable local laws and regulations.

If you feel inspired to promotethe activeasset management experience, this is the place for you. Join our international, technology-enabled, and agile environment. If you need support to navigate our websites or at any stage during your application, please send an email with your request to [e-mail].

Great to have you on board.

Let's care for tomorrow.

102748 | IT & Tech Engineering | Professional | n.a. | Allianz Global Investors | Full-Time | Permanent