Databricks Platform Engineer

Zurich Insurance Group; Madrid; Hybride; EUR 90 000 - 120 000; Plein temps

Avantages offerts par ce poste

Flexible working model

Home office allowance

Health insurance

Training programs

Stock options

Résumé du poste

Zurich Insurance Group is seeking a Databricks Platform Engineer for our Technology Delivery Center. The role focuses on owning and evolving a scalable analytics platform powering the Global Finance Data Hub, with responsibilities spanning data ingestion, Delta Lake, and governance.

You will collaborate with data engineering, architecture, and product teams to implement CI/CD, IaC, and security controls in a regulated financial services environment.

Qualifications

• 5-8 years of experience in Data Engineering, Data Platform Engineering, or Cloud Data Architecture.
• Minimum 3 years of hands-on experience owning and managing Databricks platforms in enterprise environments.
• Strong expertise in Databricks services including Unity Catalog, Delta Lake, Delta Live Tables, Workflows, and Asset Bundles.
• Experience designing and supporting batch, streaming, API, and CDC data ingestion frameworks.
• Advanced proficiency in PySpark, Python, and Databricks SQL.
• Solid knowledge of DevOps practices including Git-based development, CI/CD pipelines, automated testing, and Infrastructure as Code (CloudFormation or CDK).
• Strong AWS experience covering S3, IAM, networking, security controls, and secrets management.
• Experience with large-scale data platforms in regulated industries; financial services experience desirable.
• Databricks and/or AWS certifications preferred.
• Experience with Airflow, Kafka/MSK, and enterprise data governance frameworks.

Responsabilités

• Own and evolve the Databricks platform underpinning the Zurich Global Finance Data Hub.
• Design, build, and maintain data ingestion frameworks across batch, streaming, API, and CDC data integration patterns.
• Develop and manage Delta Live Tables pipelines with data quality controls and schema enforcement.
• Administer and optimise Delta Lake environments, including performance tuning, schema evolution, time travel, VACUUM, and OPTIMIZE.
• Implement and manage Unity Catalog governance practices (access controls, data lineage, classification, namespaces).
• Build, deploy, and support Databricks Workflows for orchestration of multi-task pipelines.
• Establish and maintain CI/CD pipelines using Git and Databricks Asset Bundles.
• Collaborate with cloud and infrastructure teams to implement IaC (AWS CloudFormation or CDK).
• Manage authentication and authorisation mechanisms (IAM roles, instance profiles, credentials rotation).
• Monitor platform health, performance, metrics, and costs using Databricks system tables, CloudWatch, Grafana.
• Define cluster policies, compute strategies, and DBU governance controls.
• Ensure platform security (PrivateLink, VPC endpoints, row/column security).
• Partner with data engineering, finance, architecture, and product teams to deliver a scalable enterprise data platform.
• Drive platform best practices, automation, and continuous improvement across the data ecosystem.

Connaissances

Data engineering

Cloud data architecture

DevOps

PySpark

Python

Databricks fundamentals

Formation

Bachelor's/ Master's in a relevant field

Outils

Databricks Unity Catalog

Delta Lake

Delta Live Tables

Databricks Workflows

Asset Bundles

AWS (S3, IAM)

Airflow

Kafka/MSK

Description du poste

We Are Waiting for You Hi there! I am Álvaro Gallego, the recruiter for this position.Desplácese hacia abajo para encontrar una descripción detallada de este trabajo y lo que se espera de los candidatos. Nice to meet you! First, a little bit about what I like the most about working at Zurich Technology Delivery Center: The environment here is incredibly collaborative and innovative. Our tech-driven mindset fosters continuous learning and growth, creating an atmosphere where everyone can thrive. Now, I can't wait to share this exciting journey with you at Zurich! First of all, THANK YOU for considering our company. We understand that finding the right opportunity can be tough, but I’ll try to make it easier.

Who Are We?

Let me introduce us. You’re reading about Zurich Insurance Group, a global leader in insurance, but this role specifically pertains to our Technology Delivery Center (TDC). Established in Barcelona in 2006, the TDC operates as an integral part of Zurich Insurance Group, providing cutting-edge technological solutions and support to Zurich's local and global business units. In essence, Zurich TDC is where technology meets the vast reach of a multinational company. We specialize in areas such as software development, data analytics, cybersecurity, and IT infrastructure. Our primary mission is to support Zurich's strategic goals and enhance operational efficiency through innovative technology solutions. If you're considering a role with us, know that you'll be joining a hub of technological excellence dedicated to driving Zurich's success on a global scale.

What Can You Expect?

And you might be thinking: This is great, what will my day-to-day be like, at TDC? I’m glad you asked. As a Databricks Platform Engineer you will have the opportunity to:

• Own and evolve the Databricks platform that underpins the Zurich Global Finance Data Hub, ensuring scalability, reliability, security, and cost efficiency.
• Design, build, and maintain data ingestion frameworks supporting batch, streaming, API, and CDC data integration patterns across multiple source systems.
• Develop and manage Delta Live Tables (DLT) pipelines, implementing data quality controls, expectations, schema enforcement, and change data capture processes.
• Administer and optimise Delta Lake environments, including performance tuning, schema evolution, time travel, VACUUM, and OPTIMIZE operations.
• Implement and manage Unity Catalog governance practices, including access controls, data lineage, data classification, and namespace management.
• Build, deploy, and support Databricks Workflows for orchestration of complex multi-task pipelines, dependency management, retries, and event-driven processing.
• Establish and maintain CI/CD pipelines using Git-based development practices and Databricks Asset Bundles to promote code across development, testing, and production environments.
• Collaborate with cloud and infrastructure teams to implement Infrastructure as Code solutions using AWS CloudFormation or CDK.
• Manage authentication and authorisation mechanisms, including AWS IAM roles, instance profiles, service principals, secret management, and credential rotation processes.
• Monitor platform health, performance, operational metrics, and costs using Databricks system tables, CloudWatch, Grafana, and related observability tools.
• Define and enforce cluster policies, compute strategies, and DBU governance controls to optimise platform utilisation and expenditure.
• Ensure platform security through the implementation of PrivateLink, VPC endpoints, row-level security, and column-level security controls.
• Partner with data engineering, finance, architecture, and product teams to deliver a robust and scalable enterprise data platform.
• Drive platform best practices, technical standards, automation, and continuous improvement initiatives across the data ecosystem.

What Makes You a Great Fit?

While we’re not seeking perfection or expecting you to save the world, we believe that if you meet some of these requirements, you’ll be a fantastic fit for our team.

• 5-8 years of experience in Data Engineering, Data Platform Engineering, or Cloud Data Architecture.
• Minimum 3 years of hands‑on experience owning and managing Databricks platforms in enterprise environments.
• Strong expertise in Databricks services, including Unity Catalog, Delta Lake, Delta Live Tables, Workflows, and Asset Bundles.
• Proven experience designing and supporting batch, streaming, API, and CDC data ingestion frameworks.
• Advanced proficiency in PySpark, Python, and Databricks SQL for scalable data pipeline development.
• Solid knowledge of DevOps practices, including Git-based development, CI/CD pipelines, automated testing, and Infrastructure as Code (CloudFormation or CDK).
• Strong AWS experience covering S3, IAM, networking, security controls, and secrets management.
• Experience working with large-scale data platforms in regulated industries; financial services experience is highly desirable.
• Preferred: Databricks and/or AWS certifications.
• Experience with Airflow, Kafka/MSK, and enterprise data governance frameworks.
• And…

And… What Makes Us Special?

As well as a competitive salary and a yearly bonus, we offer:

• To support your work‑life balance, we have adopted a flexible working model.
• Option to work abroad up to 25 days yearly.
• Over 300 euros to set up your home office and additional monthly home office allowance.
• Wide range of internal and external trainings, including English, German and Spanish classes depending on the needs.
• Ticket restaurant and Health Insurance with the flexibility to exchange it for other benefits.
• Life and accident insurance.
• Collective Life retirement Plan
• 2000 referral bonus if you bring other talented people like you to the company.
• Special banking and insurance conditions plus Exclusive Employees discounts
• Functional diversity benefits
• Stock options and mortgage benefits.

Where Are We Located?

Barcelona, Poblenou.#Li‑hybrid

At Zurich, we are an equal opportunity employer. We attract and retain the best-qualified individuals available, regardless of race/ethnicity, religion, gender, sexual orientation, age, or disability. Zurich Technology Delivery Center – Your Talent, Our Strength. You are the heart & soul of Zurich! At Zurich, we like to think outside the box and challenge the status quo. We take an optimistic approach by focusing on the positives and constantly asking What can go right? We highly value the experience and know‑how of our employees and offer a wide range of opportunities across business areas to encourage you to apply for new opportunities within Zurich when you are ready for your next career step. xiphteb Let’s continue to grow together!

Location(s):ES- Barcelona Schedule: Full TimeRecruiter name: Alvaro Gallego Zazo

Databricks Platform Engineer • Madrid, Kingdom Of Spain, ES