Data engineer

TechYard; Barcelona; Hybrid; EUR 70,000 - 110,000; Full time

Job summary

TechYard in Barcelona is seeking an experienced Databricks Data Engineer to design, build, and optimise scalable data pipelines and data platforms using Databricks, Apache Spark, and cloud technologies. You will collaborate with data architects, analysts, data scientists, and business stakeholders to transform complex datasets into reliable, high-quality data solutions that drive business value.

The role demands strong Python, PySpark, SQL skills and experience with Delta Lake and lakehouse

Qualifications

• Proven experience as a Data Engineer with Databricks.
• Strong proficiency in Python, PySpark, and SQL.
• Hands-on experience with Apache Spark and Delta Lake.
• Experience designing and implementing ETL/ELT pipelines.
• Knowledge of cloud data platforms (Azure/AWS/GCP).
• Experience with data modelling, warehousing, and lakehouse architectures.
• Familiarity with Git, CI/CD, and data orchestration.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.
• Build and optimise data processing workflows using PySpark and SQL.
• Develop and manage data lakehouse architectures using Delta Lake and the Databricks Lakehouse Platform.
• Integrate data from multiple sources, ensuring data quality, consistency, and reliability.
• Implement data transformation, orchestration, and workflow automation.
• Optimise pipeline performance, scalability and cost efficiency.
• Apply data governance, security, and access control best practices.
• Collaborate with cross-functional teams to deliver robust, production-ready data solutions.
• Monitor, troubleshoot, and improve existing data pipelines and data platforms.

Skills

Python

PySpark

SQL

Databricks

Apache Spark

ETL/ELT

Data governance

Agile

Tools

Delta Lake

Azure

AWS

Google Cloud

Git

CI/CD

Terraform

Lakehouse Platform

Job description

Описание

We are looking for an experienced Databricks Data Engineer to join our growing data team in Barcelona. In this role, you will design, build, and optimise scalable data pipelines and data platforms using Databricks, Apache Spark, and cloud technologies. You will work closely with data architects, analysts, data scientists, and business stakeholders to transform complex datasets into reliable, high-quality data solutions that drive business value.

Задачи

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.
• Build and optimise data processing workflows using PySpark and SQL.
• Develop and manage data lakehouse architectures using Delta Lake and the Databricks Lakehouse Platform.
• Integrate data from multiple sources, ensuring data quality, consistency, and reliability.
• Implement data transformation, orchestration, and workflow automation.
• Optimise pipeline performance, scalability, and cost efficiency.
• Apply data governance, security, and access control best practices.
• Collaborate with cross-functional teams to deliver robust, production-ready data solutions.
• Monitor, troubleshoot, and improve existing data pipelines and data platforms.

Требования

• Proven experience as a Data Engineer working with Databricks.
• Strong proficiency in Python, PySpark, and SQL.
• Hands-on experience with Apache Spark and Delta Lake.
• Experience designing and implementing ETL/ELT data pipelines.
• Knowledge of cloud data platforms, preferably Azure, AWS, or Google Cloud.
• Experience with data modelling, data warehousing, and lakehouse architectures.
• Familiarity with Git, CI/CD, and data pipeline orchestration.
• Strong analytical, problem-solving, and communication skills.
• Ability to work effectively within an Agile, collaborative environment.

Будет плюсом

• Experience with Azure Databricks and Microsoft Azure data services
• Knowledge of Delta Live Tables / Lakeflow Declarative Pipelines and Unity Catalog
• Experience with Terraform, infrastructure as code, or automated deployment practices
• Familiarity with Power BI, data analytics, or machine learning workflows
• Relevant Databricks or cloud certifications

Условия

• Hybrid working model based in Barcelona.
• Opportunity to work on modern data engineering and cloud transformation projects.
• Collaborative and international working environment.
• Opportunities for professional development and technical growth.
• Competitive compensation package, aligned with experience.