Hibrido BCN - Senior Data Platform Engineer

Capitole Consulting; Barcelona; On-site; EUR 90,000 - 130,000; Full time

Benefits offered by this job

Professional development allowance (€1

Private health insurance

Meal and transport vouchers

Capitole Club discounts

Wellhub subscription

Team events and sports clubs

Job summary

Capitole Consulting is seeking a Senior Data Platform Engineer to evolve a large Azure-based data platform and lakehouse architecture. You will own foundations used by data product teams, including Airflow orchestration, Terraform-based infrastructure, and Unity Catalog governance, while staying hands-on and close to the consuming teams.

The role focuses on platform reliability, cost efficiency, and observability, with leadership in architectural reviews and engineering standards.

Qualifications

• 6+ years of data engineering experience focusing on platform/infrastructure topics.
• Strong Python and SQL (advanced) with PySpark and Delta Lake semantics.
• Hands-on production experience with Apache Airflow.
• Hands-on Azure resource management, including IaC with Terraform.
• Experience with Databricks and Unity Catalog governance.
• Familiarity with cost management and governance of cloud resources.

Responsibilities

• Own the Airflow platform: environments, deployment, upgrades, configuration and DAG standards.
• Migrate from Azure Data Factory to Airflow and evolve the orchestration framework.
• Ensure Airflow is reliable and observable with monitoring and alerts.
• Manage Azure resources with Terraform: Databricks, Data Lake Storage, Key Vault, networking.
• Handle secrets/identity and access management (Key Vault, Azure AD, Unity Catalog).
• Oversee Azure DevOps pipelines, multi-environment promotion and Terraform state.
• Contribute to Data Catalog/Quality framework and overall observability across the platform.

Skills

Python

SQL

Data engineering

Cross-functional collaboration

Problem solving

Tools

Apache Airflow

Terraform

Databricks

Unity Catalog

Azure DevOps

Key Vault

Azure RBAC

Job description

Capitole is growing fast, and we want you to be part of the journey!

We are looking for a Senior Data Platform Engineer with +6 years of experience to help drive the evolution of our Azure-based Data Platform and Lakehouse architecture. This role is focused on the platform itself. The professional will own and evolve the foundations that data product teams build on: Apache Airflow orchestration (migrating from ADF), Azure resource management (Key Vault, identity and access, networking, storage), Azure DevOps CI/CD and Infrastructure as Code, Databricks and Unity Catalog governance, observability and cost efficiency, while remaining hands‑on and close to the teams that use the platform.

Responsibilities:

Orchestration Platform (Apache Airflow & ADF)

• Own the Apache Airflow platform: environments, deployment, upgrades, configuration, connections, variables, pools and DAG standards.
• Work on the migration from Azure Data Factory to Airflow and the evolution of the Metadata-Driven Orchestration Framework (DAG generation, reusable operators, dependency and scheduling patterns).
• Ensure Airflow is reliable and observable: SLAs, alerting, retries, backfills, health monitoring and clear failure ownership.

Azure Resource Management

• Manage Azure resources with Infrastructure as Code (Terraform): Databricks workspaces, Data Lake Storage, Data Factory, Key Vault, networking and supporting services.
• Own secrets and identity management: Key Vault, service principals and managed identities, secret scopes and rotation.
• Own access management: Azure RBAC, Entra ID groups, Unity Catalog grants and ownership, following least privilege, plus access requests and periodic reviews.
• Own Azure DevOps: repositories, CI/CD pipelines, branch policies and multi‑environment promotion, including deployment validation and Terraform state.
• Track cost and governance of Azure resources: tagging, budgets, right‑sizing and cost attribution (e.g. query tagging).

Hands‑on Data Platform evolution

• Contribute to the implementation of the Data Catalog and Data Quality Framework and it’s lineage across the platform.
• Drive observability: pipelines that are observable, debuggable and alert‑driven, with clear ownership of failures.
• Operate the Databricks platform: Unity Catalog structure, cluster and job policies, Auto Loader ingestion patterns, and performance and cost optimization of workloads.

Technical Leadership

• Participate in architecture reviews and technical decision‑making, stating trade offs (performance, cost, reliability, complexity) with data.
• Define and promote engineering standards: naming conventions, patterns and guidelines, through code reviews and design reviews.
• Collaborate with Architects, Product Owners and data engineers in technical maturation of data products.
• Build and maintain developer tooling and automation, including AI‑assisted engineering workflows (shared agent instructions, skills, automated PR review).

Support & Operations (Run Team Collaboration)

• Provide second‑line support to the Run/Operations team for complex production topics.
• Troubleshoot and resolve platform incidents, failures and performance issues; lead root cause analysis and post‑mortems.
• Ensure proper handover, documentation and operational readiness of platform changes.
• Continuously improve runbooks, monitoring and operational processes.

Requirements:

Technical Skills

• 6+ years of experience in data engineering, with significant time on platform or infrastructure topics
• Strong technical expertise with Python and SQL (advanced), including PySpark and Delta Lake semantics
• Deep hands‑on experience with Apache Airflow in production: DAG design, deployment, operations, upgrades, and orchestration of Databricks and ADF workloads
• Hands‑on experience with Azure resource management:
  • Key Vault, managed identities and service principals, secret management
  • Access management: Azure RBAC, Entra ID, Unity Catalog permissions
  • Azure DevOps (repos, pipelines, service connections, environments) and CI/CD for data platforms
  • ADLS, networking and security basics, Azure Data Factory
• Infrastructure as Code (Terraform) and multi‑environment promotion
• Hands‑on experience with Databricks and Unity Catalog (compute, jobs, governance)
• Observability and monitoring, alerting and incident management
• Solid understanding of Data Lakehouse architecture, Medallion architecture and metadata‑driven architecture; awareness of cost management and performance

Soft Skills

• Strong problem‑solving and analytical thinking
• Ability to work in cross‑functional teams and to influence without authority
• Ownership mindset and attention to detail
• Good communication skills with both technical and business stakeholders: leads with the recommendation, quantifies, and is transparent about uncertainty

What we’re looking for in you:

• Led an Airflow implementation or migration (ideally from ADF), or run it at scale.
• Built metadata‑driven orchestration or reusable ETL frameworks.
• Built developer tooling or LLM-agent workflows (MCP, skills, automated review) for engineering teams.
• Applied FinOps practices on Azure and Databricks: cost attribution, budgets, forecasting and measurable savings
• Experience running Airflow on Kubernetes or operating containerized workloads on AKS

Why you’ll love working with us

At Capitole, we believe our people are our greatest strength. That's why we offer:

• €1,200/year for your professional development (trainings, events, certifications…)
• Monthly feedback sessions with your team to keep learning and growing
• Private health insurance fully covered by Capitole
• Payflow benefits (meal, transport, childcare vouchers)
• Wellhub subscription
• Discounts with top brands through Capitole Club

Life at Capitole

• Team buildings every two months (our summer party and Christmas dinner are legendary!)
• Football, padel, and basketball teams sponsored by Capitole
• Tech communities where sharing knowledge is part of our DNA
• And above all… an AMAZING TEAM that you’ll love being a part of!

Want to learn more about us? capitole-consulting.com

See what our teams are saying about us: Glassdoor Reviews

• Will have access to confidential information related to Capitole and the project they are working on.
• Must comply with the security and internal policies of both the company and the client.
• Must sign an NDA (Non-Disclosure Agreement).