Databricks Platform Engineer — Enterprise Data Hub

Tamarind Intelligence; Barcelona; Híbrido; EUR 55.000 - 90.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flexible working model

Travel up to 25 days yearly

Home office allowance

Language trainings

Health insurance

Life and retirement plans

Stock options

Descripción de la vacante

Zurich Insurance Group's Technology Delivery Center in Barcelona seeks a data platform engineer to own and evolve our Databricks-based data hub. You will design and operate Delta Lake pipelines, Delta Live Tables, and Unity Catalog governance while partnering with cloud and infrastructure teams to implement IaC using AWS CDK/CDK.

The role offers a competitive salary, yearly bonus, flexible working model, and opportunities for international collaboration across Zurich's global units.

Formación

• Experience with Databricks platform and Delta Lake architecture.
• Knowledge of data ingestion pipelines (batch/streaming/API).
• Familiarity with CI/CD for data platforms and orchestration.
• Certifications in Databricks or AWS preferred.
• Experience with AWS IAM and security controls.

Responsabilidades

• Own and evolve the Databricks platform that underpins the Zurich Global Finance Data Hub, ensuring scalability, reliability, security, and cost efficiency.
• Design, build, and maintain data ingestion frameworks supporting batch, streaming, API, and CDC data integration patterns across multiple source systems.
• Develop and manage Delta Live Tables (DLT) pipelines, implementing data quality controls, expectations, schema enforcement, and change data capture processes.
• Administer and optimise Delta Lake environments, including performance tuning, schema evolution, time travel, VACUUM, and OPTIMIZE operations.
• Implement and manage Unity Catalog governance practices, including access controls, data lineage, data classification, and namespace management.
• Build, deploy, and support Databricks Workflows for orchestration of complex multi-task pipelines, dependency management, retries, and event-driven processing.
• Establish and maintain CI/CD pipelines using Git-based development practices and Databricks Asset Bundles to promote code across development, testing, and production environments.
• Collaborate with cloud and infrastructure teams to implement Infrastructure as Code solutions using AWS CloudFormation or CDK.
• Manage authentication and authorisation mechanisms, including AWS IAM roles, instance profiles, service principals, secret management, and credential rotation processes.
• Monitor platform health, performance, operational metrics, and costs using Databricks system tables, CloudWatch, Grafana, and related observability tools.
• Define and enforce cluster policies, compute strategies, and DBU governance controls to optimise platform utilisation and expenditure.
• Ensure platform security through the implementation of PrivateLink, VPC endpoints, row-level security, and column-level security controls.
• Partner with data engineering, finance, architecture, and product teams to deliver a robust and scalable enterprise data platform.
• Drive platform best practices, technical standards, automation, and continuous improvement initiatives across the data ecosystem.

Conocimientos

Databricks

AWS

Airflow

Kafka/MSK

Data governance

Herramientas

Delta Live Tables

Delta Lake

Unity Catalog

CloudFormation/CDK

CI/CD tooling

Descripción del empleo

Zurich Insurance Group's Technology Delivery Center in Barcelona seeks a data platform engineer to own and evolve our Databricks-based data hub. You will design and operate Delta Lake pipelines, Delta Live Tables, and Unity Catalog governance while partnering with cloud and infrastructure teams to implement IaC using AWS CDK/CDK.

The role offers a competitive salary, yearly bonus, flexible working model, and opportunities for international collaboration across Zurich's global units.