Databricks Data Engineer

TechShack; España; Remote; EUR 78,000 - 106,000; Full time

Job summary

TechShack is seeking an experienced Databricks Data Engineer for a 12+ month remote contract. The role focuses on migrating legacy workloads to a modern Lakehouse on Databricks, designing scalable data pipelines and establishing engineering standards across the platform.

You will collaborate with architects, analysts and insurance stakeholders, delivering production-grade PySpark and SQL solutions while enabling downstream BI, analytics and future AI/ML use cases.

Qualifications

• 5+ years of data engineering experience.
• 2+ years of hands-on Databricks delivery in production.
• Proficient in Python (focus on PySpark) and SQL.
• Experience with data modelling (dimensional/Data Vault).
• Cloud experience, ideally Azure (AWS also considered).
• CI/CD and IaC practices for data platforms.

Responsibilities

• Design, build and maintain scalable batch and streaming pipelines on Databricks using PySpark and Delta Lake.
• Migrate legacy ETL/ELT workloads to the Lakehouse architecture.
• Implement Medallion patterns and reusable ingestion/transformation frameworks.
• Orchestrate workflows with Databricks Workflows, Delta Live Tables, Azure Data Factory or Airflow.
• Contribute to CI/CD and infrastructure as code using Databricks Asset Bundles, Terraform, Azure DevOps or GitHub Actions.

Skills

5+ years data engineering

2+ years Databricks delivery

Advanced Python

Advanced PySpark

Advanced SQL

Data modelling

Communication

CI/CD for data engineering

Tools

Databricks

Azure Data Factory

GitHub Actions

Terraform

Delta Lake

Airflow

dbt

Job description

Databricks Data Engineer | 12 Month+ B2B Contract | Fully Remote | €500 per day

I'm partnering with a global technology consultancy on a large data transformation programme for a well-established organisation in the insurance sector. They need an experienced, hands‑on Databricks Data Engineer to join the delivery team on a long‑term contract.

The Role

The end client is moving away from legacy data warehousing and scattered on‑premise systems to a modern, unified Lakehouse platform built on Databricks. You will sit at the centre of this transformation. You'll design and build scalable data pipelines, migrate legacy workloads, and help set the engineering standards the wider platform will be built on.

You’ll work closely with architects, analysts and business stakeholders across the underwriting, claims, policy and finance data domains.

This is a hands‑on engineering role. You should be comfortable writing production‑grade PySpark and SQL every day.

The Project

• Migrating legacy data warehouse and ETL workloads (SQL Server, SSIS, Informatica or similar) onto Databricks
• Designing and building a Medallion (Bronze, Silver, Gold) Lakehouse architecture on Delta Lake
• Rolling out Unity Catalog for central governance, lineage and access control
• Delivering curated, trusted data products for actuarial, risk, claims and regulatory reporting
• Introducing CI/CD, automated testing and infrastructure as code across the platform
• Enabling downstream BI, analytics and future AI/ML use cases

Key Responsibilities

• Design, build and maintain scalable batch and streaming pipelines on Databricks using PySpark, Spark SQL and Delta Lake
• Migrate legacy ETL processes and data models onto the Lakehouse, keeping data accurate and reconciled throughout
• Implement Medallion architecture patterns and build reusable ingestion and transformation frameworks
• Orchestrate workflows using Databricks Workflows, Delta Live Tables, Azure Data Factory or Airflow
• Work within Unity Catalog to enforce governance, security and lineage
• Optimise Spark jobs and cluster configurations for performance and cost
• Build data quality checks, monitoring and alerting into pipelines
• Contribute to CI/CD and infrastructure as code using Databricks Asset Bundles, Terraform, Azure DevOps or GitHub Actions
• Turn business requirements into robust data solutions alongside architects, analysts and insurance stakeholders
• Document solutions and contribute to engineering best practice across the programme
• 5+ years of commercial experience as a Data Engineer
• 2+ years of hands‑on Databricks delivery in production
• Advanced Python and PySpark
• Advanced SQL and data modelling (dimensional, Data Vault or similar)
• Delta Lake and Lakehouse / Medallion architecture
• Cloud platform experience, ideally Azure (AWS also considered)
• Building and orchestrating ETL/ELT pipelines at scale
• Git-based workflows and CI/CD for data engineering
• Experience on a data migration or transformation programme
• Confident communicator, comfortable working on your own in a remote team
• Unity Catalog implementation or migration
• Delta Live Tables and Structured Streaming
• Databricks Asset Bundles and Terraform
• Insurance, financial services or other regulated industry experience
• Insurance data domains such as policy, claims, underwriting and actuarial
• Exposure to regulatory reporting (Solvency II, IFRS 17)
• dbt, Azure Data Factory or Airflow
• Data quality frameworks such as Great Expectations or DLT expectations
• Previous consultancy or client‑facing delivery experience

Certification

The ideal candidate holds the Databricks Certified Data Engineer Professional certification, and certified candidates will be prioritised. Associate‑certified engineers with strong hands‑on delivery experience are also encouraged to apply.

Why Apply?

• 12 month+ initial contract with a strong chance of extension
• Fully remote working
• €500 per day on a B2B basis
• Work on a near‑greenfield build on a modern Databricks Lakehouse platform
• Real influence over engineering standards and architecture as the platform takes shape
• A high‑profile programme with a global consultancy and a major insurance client

Please note:

this role is not suited to purely BI or reporting profiles, or to architects and delivery leads who no longer code day to day. Candidates must be able to work on a B2B contract basis.