Databricks Data Engineer - End-to-End Pipelines & Medallion

JobServe Ltd; Madrid; Hybrid; EUR 48,000 - 63,000; Full time

Job summary

JobServe Ltd is seeking a hands-on Databricks Data Engineer to design and implement end-to-end data pipelines in Madrid, with two days on site weekly. This contract role focuses on building scalable pipelines using PySpark, Python and SQL within a Medallion architecture.

You will ingest data from APIs, databases and files, optimise performance and ensure data quality across Bronze, Silver and Gold layers, delivering reliable data for analytics.

Qualifications

• Strong hands-on experience with Databricks.
• Advanced PySpark development skills.
• Strong Python programming experience.
• Proven experience building end-to-end data pipelines.
• Expertise in data ingestion from multiple sources, including APIs, databases and files.
• Experience implementing Medallion architecture (Bronze, Silver and Gold layers).
• Strong SQL skills and experience in data transformation.
• Experience handling large-scale data processing.

Responsibilities

• Design and develop end-to-end data pipelines from source systems to Gold layer datasets.
• Implement data ingestion, transformation and optimisation using Databricks.
• Build scalable data processing pipelines using PySpark.
• Ingest data from multiple sources, including APIs, databases and files.
• Implement Medallion architecture across Bronze, Silver and Gold layers.
• Develop and optimise data transformations using SQL and Python.
• Process large-scale datasets with data quality and reliability.
• Ensure pipelines are maintainable, scalable and performant.

Skills

Databricks

PySpark

Python

SQL

Medallion architecture

Data pipelines

API ingestion

Databases

Files ingestion

Job description

JobServe Ltd is seeking a hands-on Databricks Data Engineer to design and implement end-to-end data pipelines in Madrid, with two days on site weekly. This contract role focuses on building scalable pipelines using PySpark, Python and SQL within a Medallion architecture.

You will ingest data from APIs, databases and files, optimise performance and ensure data quality across Bronze, Silver and Gold layers, delivering reliable data for analytics.