Azure Databricks Data Engineer — Streaming & Scale

STAFIDE; Amsterdam; On-site; EUR 110,000 - 140,000; Full time

Job summary

STAFIDE is seeking an experienced Azure Databricks Engineer to design, develop, and maintain data pipelines using Azure Databricks, Python, and PySpark. You will build batch and streaming workflows, design data models, and ensure performance in a large-scale cloud environment.

You should have 8+ years in data or software engineering, deep Azure Databricks expertise, and hands-on experience with CI/CD pipelines (_yaml_) and IaC (ARM/Bicep).

Qualifications

• 8+ years of overall professional experience in data engineering/software engineering.
• Strong hands-on experience with Azure Databricks as a core technology.
• Deep expertise in Python, PySpark, and SQL.
• Proven experience designing and managing data processing solutions within Databricks.
• Strong experience handling both streaming and batch data.
• Experience designing data models specifically for Databricks-based processing environments.
• Solid experience with Microsoft Azure cloud services.
• Hands-on experience with CI/CD pipelines using YAML.
• Experience with Infrastructure as Code, particularly ARM/Bicep.
• Experience working in an Agile environment.

Responsibilities

• Design, develop, and maintain robust and reliable data processing solutions using Azure Databricks, Python, and PySpark.
• Build and manage both batch and streaming data processing workflows within Databricks.
• Design appropriate data models and processing architectures within the Databricks environment.
• Manage multiple parallel data processes working on shared source data.
• Develop scalable and maintainable data solutions while ensuring high standards of quality, reliability, and performance.
• Implement and maintain CI/CD pipelines using Azure and YAML.
• Work with Infrastructure as Code (IaC) using ARM/Bicep.
• Continuously improve existing solutions, reduce technical debt, and make pragmatic technical decisions.
• Collaborate effectively within an Agile engineering environment.

Skills

Azure Databricks

Python

PySpark

SQL

CI/CD pipelines

YAML

Infrastructure as Code

Agile methodologies

Tools

ARM/Bicep

Azure cloud services

Job description

STAFIDE is seeking an experienced Azure Databricks Engineer to design, develop, and maintain data pipelines using Azure Databricks, Python, and PySpark. You will build batch and streaming workflows, design data models, and ensure performance in a large-scale cloud environment.

You should have 8+ years in data or software engineering, deep Azure Databricks expertise, and hands-on experience with CI/CD pipelines (_yaml_) and IaC (ARM/Bicep).