Senior Azure Databricks Engineer — Scalable Data Pipelines

STAFIDE; Amsterdam; On-site; EUR 90,000 - 130,000; Full time

Job summary

STAFIDE is seeking a Senior Azure Databricks Engineer to design, develop, and optimize scalable data processing solutions in Azure Databricks. You will build batch and streaming pipelines, craft efficient PySpark transformations, and model data for high performance.

You will work in an Agile environment with cross-functional teams, implement CI/CD using Azure DevOps, and apply IaC with ARM/Bicep to automate deployments. Strong communication and troubleshooting skills are essential.

Qualifications

• Strong hands-on experience with Azure Databricks as a core data engineering platform.
• Strong proficiency in Python, PySpark, and SQL.
• Hands-on experience developing batch and streaming data pipelines.
• Experience with data modeling, transformation, and optimization within Databricks environments.
• Good understanding of Azure cloud services relevant to data engineering.
• Experience with Azure DevOps, CI/CD, and YAML-based pipeline configurations.
• Hands-on experience with Infrastructure as Code, particularly ARM/Bicep.
• Experience working in Agile engineering and delivery environments.
• Understanding of modern cloud-based data architectures and end-to-end data engineering solutions.
• Strong communication, collaboration, troubleshooting, and problem-solving skills.

Responsibilities

• Design, develop, and maintain scalable data processing solutions using Azure Databricks.
• Build and manage robust batch and streaming data pipelines within Databricks environments.
• Develop and optimize data transformations using Python, PySpark, and SQL.
• Design and optimize data models to support scalable processing, performance, and reliability.
• Manage multiple parallel data processing workflows and shared data sources.
• Implement and maintain CI/CD pipelines using Azure DevOps and YAML-based configurations.
• Apply Infrastructure as Code using ARM/Bicep for deployment and automation.
• Monitor, troubleshoot, and optimize data processing workloads and Databricks environments.
• Collaborate with cross-functional engineering and business teams to deliver reliable data solutions.
• Contribute to Agile development practices and maintain high engineering standards.

Skills

Azure Databricks

Python

PySpark

SQL

Batch & streaming pipelines

Data modeling

Azure cloud services

Azure DevOps

CI/CD

YAML

IaC ARM/Bicep

Agile

Cloud architecture

Communication & collaboration

Tools

Azure Databricks

Azure DevOps

ARM/Bicep

YAML

Job description

STAFIDE is seeking a Senior Azure Databricks Engineer to design, develop, and optimize scalable data processing solutions in Azure Databricks. You will build batch and streaming pipelines, craft efficient PySpark transformations, and model data for high performance.

You will work in an Agile environment with cross-functional teams, implement CI/CD using Azure DevOps, and apply IaC with ARM/Bicep to automate deployments. Strong communication and troubleshooting skills are essential.