Senior Databricks Data Engineer — Remote from Spain

Intellias; Mos; Presencial; EUR 90.000 - 130.000; Jornada completa

Descripción de la vacante

Intellias is hiring a Senior Data Engineer to lead Databricks-based data platform migration for a financial services client. You will own architectural decisions, mentor a mid-level engineer, and deliver end-to-end data domain pipelines on Azure Databricks.

The role emphasizes hands-on coding, production-grade pipelines, and implementing data quality, lineage, and access controls, with modern CI/CD and AI-enabled tooling to accelerate delivery.

Formación

• 6+ years of data engineering experience, with at least the last three years on Databricks in production.
• Deep hands-on experience with medallion / Lakehouse architecture, Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines and Databricks Asset Bundles.
• Strong python, PySpark and SQL, comfortable diagnosing performance issues on large workloads through Spark UI and Photon.
• Real production experience with streaming ingestion (Kafka Structured Streaming, Auto Loader) and CDC patterns.
• Experience implementing data quality at scale (Great Expectations or equivalent), plus lineage, cataloguing and access control.
• CI/CD on data platforms through Azure DevOps or GitHub Actions, with infrastructure as code (Terraform or equivalent).
• Fluent working English.

Responsabilidades

• Perform the end-to-end delivery of one or more priority data domains on Azure Databricks.
• Design and build medallion (Bronze / Silver / Gold) pipelines using Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming and Delta Lake with Liquid Clustering.
• Register curated data products in Unity Catalog with the correct tags, masks, row filters, lineage and access policies.
• Implement data quality gates at the Silver-to-Gold boundary, and refine rules with domain stewards.
• Package and deploy work using Databricks Asset Bundles through Azure DevOps CI/CD
• Write automated tests and documentation
• Work at AI-as-Collaborator level today with a path to AI-as-Orchestrator, using Claude Code, Databricks Assistant, GitHub Copilot and MCP-connected agent tooling to accelerate the recurring parts of pipeline build, test and deploy.

Conocimientos

Databricks

Python

PySpark

SQL

Delta Lake

Unity Catalog

Lakeflow Declarative Pipelines

CI/CD

Azure DevOps

GitHub Actions

Terraform

Spark UI

Photon

Kafka Structured Streaming

Great Expectations

Data quality

Data governance

MCP-connected tooling

Herramientas

Databricks

Azure DevOps

GitHub Actions

Terraform

Unity Catalog

Lakeflow

Delta Lake

Descripción del empleo

Intellias is hiring a Senior Data Engineer to lead Databricks-based data platform migration for a financial services client. You will own architectural decisions, mentor a mid-level engineer, and deliver end-to-end data domain pipelines on Azure Databricks.

The role emphasizes hands-on coding, production-grade pipelines, and implementing data quality, lineage, and access controls, with modern CI/CD and AI-enabled tooling to accelerate delivery.