Senior Cloud Data Engineer: Azure Databricks & Spark

Levy Professionals; Amsterdam; On-site; EUR 90,000 - 130,000; Full time

Job summary

Levy Professionals is seeking a Senior Data Engineer to join an Agile Data Engineering team and help build a modern, cloud-based data platform. You will design, develop, and operate scalable data solutions that support critical business processes and enable data-driven decision-making.

The role emphasizes design, development, and operational excellence across data pipelines, governance, and DevOps practices in a fast-paced environment.

Qualifications

• 5+ years of experience in Data Engineering, preferably on cloud-based data platforms.
• Strong hands-on experience with Azure Databricks, including Unity Catalog and Databricks Asset Bundles (DAB).
• Hands-on experience with Apache Airflow, ADLS, and Azure Functions.
• Strong proficiency in Python (OOP), PySpark, and SQL.
• Solid understanding of Apache Spark performance tuning and large-scale data processing.
• Experience building and maintaining ETL/ELT pipelines and modern data integration solutions.
• Experience with Git, CI/CD, and Azure DevOps.
• Knowledge of data modelling techniques, including dimensional modelling.
• Strong analytical, troubleshooting, and communication skills.
• Experience working in Agile/Scrum and DevOps environments.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data products using Azure Databricks, PySpark, Python, and SQL.
• Build and optimize data ingestion, transformation, and storage solutions for analytics, reporting, and operational use cases.
• Develop and support ETL/ELT processes integrating data from multiple sources into trusted and governed data platforms.
• Design and review cloud-native solutions leveraging ADLS, Azure Functions, Databricks, and Apache Airflow.
• Ensure data quality, reliability, scalability, monitoring, and operational excellence across the data ecosystem.
• Implement and maintain source control, CI/CD pipelines, and deployment automation using Azure DevOps.
• Collaborate with business and technical stakeholders to translate requirements into robust and sustainable solutions.
• Enhance existing data products, optimize platform capabilities, and continuously improve engineering practices.
• Troubleshoot and resolve complex production issues using strong analytical and problem-solving skills.
• Contribute to data architecture, engineering standards, and best practices across the organization.
• Promote innovation, knowledge sharing, and continuous improvement within the Data Engineering community.
• Champion Agile, DevOps, and DataOps principles as part of a cross-functional Scrum team.

Skills

Data engineering

Azure Databricks

Python

PySpark

SQL

Apache Spark

Git

CI/CD

Azure DevOps

Agile/Scrum

Tools

Apache Airflow

Delta Lake

Iceberg

Unity Catalog

Databricks Asset Bundles

Job description

Levy Professionals is seeking a Senior Data Engineer to join an Agile Data Engineering team and help build a modern, cloud-based data platform. You will design, develop, and operate scalable data solutions that support critical business processes and enable data-driven decision-making.

The role emphasizes design, development, and operational excellence across data pipelines, governance, and DevOps practices in a fast-paced environment.