Senior Azure Data Engineer Databricks Pipelines & Streaming

Atrium (EMEA); Dublin; On-site; EUR 90,000 - 120,000; Full time

Job summary

Atrium (EMEA) in Dublin seeks a hands-on Senior Azure Data Engineer to design, build and maintain scalable data pipelines using Databricks (PySpark) and Azure Data Lake. You will work on event-based streaming ingestion and optimize Spark workloads, collaborating with leads and peers to deliver robust data solutions.

The role requires 6–9 years in data engineering, strong SQL and Python, and a pragmatic, delivery-oriented mindset to ensure quality and timely delivery in a production environment.

Qualifications

• 6–9 years of experience in data engineering roles.
• Hands-on experience with PySpark, SQL and Python.
• Experience building and optimizing data pipelines on Azure.
• Strong understanding of ETL/ELT best practices and data modeling.

Responsibilities

• Design, build, and maintain scalable data pipelines using Azure Databricks (PySpark) and Azure Data Lake.
• Implement event-based or streaming ingestion pipelines (Event Hub, Kafka, etc.).
• Develop reliable ETL/ELT workflows with data quality and performance in mind.
• Optimize Spark workloads for simplicity, efficiency, and maintainability.
• Collaborate with leads to review designs and improve engineering practices.
• Participate in peer code reviews and troubleshoot production issues.

Skills

PySpark

SQL

Python

Azure Databricks

Azure Data Lake

ETL/ELT

Data pipelines

Streaming

ADF

Kafka

Event Hub

Orchestration tools

Tools

ADF

Kafka

Event Hub

Job description

Atrium (EMEA) in Dublin seeks a hands-on Senior Azure Data Engineer to design, build and maintain scalable data pipelines using Databricks (PySpark) and Azure Data Lake. You will work on event-based streaming ingestion and optimize Spark workloads, collaborating with leads and peers to deliver robust data solutions.

The role requires 6–9 years in data engineering, strong SQL and Python, and a pragmatic, delivery-oriented mindset to ensure quality and timely delivery in a production environment.