Senior Azure Data Engineer (Databricks)

Atrium (EMEA); Dublin; On-site; EUR 90,000 - 120,000; Full time

Job summary

Atrium (EMEA) in Dublin is seeking a hands-on Senior Azure Data Engineer to design, build and maintain scalable data pipelines using Azure Databricks (PySpark) and Azure Data Lake.

The role focuses on delivering reliable ETL/ELT processes, optimizing Spark workloads, and collaborating with cross-functional teams in Ireland and the US to ensure data quality and operational excellence.

Qualifications

• 6–9 years of experience in data engineering roles.
• Strong hands-on experience with PySpark, Spark, SQL, Python.
• Experience with ETL/ELT, data modeling, and distributed data processing.

Responsibilities

• Design, build, and maintain scalable data pipelines on Azure Databricks and Data Lake.
• Implement event-based or streaming ingestion pipelines (Event Hub, Kafka).
• Develop reliable ETL/ELT workflows with data quality and performance in mind.
• Optimize Spark workloads for simplicity, efficiency, and maintainability.
• Collaborate with Data Engineering Lead and peers to improve engineering practices.
• Participate in peer code reviews and troubleshoot production issues.

Skills

PySpark

SQL

Python

ETL/ELT

Data pipelines

Streaming

Data modeling

Distributed processing

Communication

Collaborative work

Tools

Azure Databricks

Azure Data Lake

Azure Data Factory

Kafka

Event Hub

Spark

Job description

Hands-on Senior Azure Data Engineer (Databricks) with 6–9 years’ experience building scalable data pipelines and streaming solutions using Databricks, PySpark and Azure Data Lake. Strong in SQL, Python and ETL/ELT, with a focus on simplicity, quality and delivery.

Role Overview:

• Location: Dublin
• Contract Type: 12 months

JOB SUMMARY:

• We are seeking a Senior Data Engineer to join our data engineering team. This is a hands-on delivery role focused on building, optimizing, and maintaining scalable data pipelines using the Azure data platform.
• The ideal candidate is pragmatic, delivery-oriented, and collaborative, someone who can design fit-for-purpose solutions, work closely with leads and peers, and contribute positively to team culture.

Required Experience & Skills

• 6–9 years of experience in data engineering roles.

Strong hands-on experience with:

• PySpark
• ADF / orchestration tools
• Solid programming experience in SQL and python.
• Good understanding of ETL/ELT best practices, data modeling, and distributed data processing.
• Experience working in production-grade data platforms.
• Strong communication skills and a collaborative, professional working style.
• Ability to take feedback positively and adapt solutions to team standards.

KEY RESPONSIBILITIES:

• Design, build, and maintain scalable data pipelines using Azure Databricks (PySpark) and Azure Data Lake.
• Hands on implementation experience in event-based or streaming ingestion pipelines (Event Hub, Kafka, etc.).
• Develop reliable ETL/ELT workflows, ensuring data quality, performance, and operational stability.
• Optimize Spark workloads with a focus on simplicity, efficiency, and maintainability.
• Work closely with the Data Engineering Lead to implement agreed designs, review solutions, and continuously improve engineering practices.
• Participate in peer code reviews, applying best practices and constructive feedback.
• Troubleshoot production issues proactively and support operational excellence.
• Collaborate effectively with cross-functional teams across Ireland and the US, communicating clearly and professionally.