Senior Azure Databricks Engineer at FalconSmartIT

FalconSmartIT; Dublin; On-site; EUR 120,000 - 180,000; Part time

Job summary

FalconSmartIT in Dublin, Ireland is seeking a Senior Azure Databricks Engineer to design, build, and optimize enterprise data platforms and pipelines. You will leverage Azure Databricks, Spark, PySpark, SQL, and Delta Lake to support analytics, reporting, and AI/ML use cases.

You will partner with data architects and business stakeholders to deliver scalable, secure data solutions, while mentoring junior engineers and contributing to CI/CD and IaC efforts in a fast-paced, enterprise environment.

Qualifications

• :

Responsibilities

• Design scalable data pipelines using Azure Databricks.
• Develop PySpark, Spark SQL, and Python applications for large-scale data processing.
• Build batch and near-real-time data pipelines.
• Implement Delta Lake architectures and lakehouse patterns.
• Develop data ingestion and transformation pipelines with ADF and Databricks.
• Work with ADLS Gen2 for storage.
• Design data models for analytics and reporting workloads.
• Implement data quality, validation, monitoring, and error handling.
• Optimize Spark jobs, clusters, and Delta tables.
• Implement Unity Catalog for data governance.
• Develop and manage Databricks workflows and production deployments.
• Set up CI/CD pipelines using Azure DevOps or GitHub Actions.
• Monitor pipelines and troubleshoot performance and data quality issues.
• Collaborate on data engineering standards and best practices.
• Mentor junior engineers and participate in code reviews.

Skills

Azure Databricks

PySpark

Spark SQL

Python

SQL

Delta Lake

ADLS Gen2

Azure Data Factory

Unity Catalog

CI/CD

Terraform (IaC)

Education

Bachelor's or Master's in CS/Engineering/IT

Tools

Databricks

Delta Lake

Azure Data Factory

ADLS Gen2

Azure DevOps

GitHub Actions

Terraform

Kafka/Event Hubs

Job description

Job Title:

Senior Azure Databricks Engineer

Job Location:

Dublin, Ireland (Onsite)

Job Type:

Contract

Role Overview

We are looking for an experienced Azure Databricks Engineer with 7+ years of hands-on experience in data engineering to join our data platform team in Dublin. The successful candidate will be responsible for designing, developing, and optimizing scalable enterprise data platforms and pipelines using Azure Databricks, Apache Spark, PySpark, SQL, Azure Data Lake Storage, and Azure Data Factory. You will work closely with data architects, analysts, application teams, and business stakeholders to deliver high-quality data solutions supporting analytics, reporting, and AI/ML use cases.

Key Responsibilities

• Design and develop scalable data pipelines and data engineering solutions using Azure Databricks.


• Develop high-performance PySpark, Spark SQL, and Python applications for large-scale data processing.


• Build and maintain batch and near-real-time data pipelines.


• Implement Delta Lake based data architectures and medallion/lakehouse patterns.


• Develop data ingestion and transformation pipelines using Azure Data Factory (ADF) and Databricks.


• Work with Azure Data Lake Storage Gen2 (ADLS Gen2) for enterprise-scale data storage.


• Design and implement data models for analytical and reporting workloads.


• Implement data quality, validation, reconciliation, monitoring, and error-handling frameworks.


• Optimize Spark jobs, cluster configurations, SQL queries, and Delta tables for performance and cost.


• Implement data governance and access controls using Unity Catalog.


• Develop and manage Databricks workflows/jobs and production deployments.


• Implement CI/CD pipelines using Azure DevOps/GitHub Actions and Infrastructure as Code where applicable.


• Monitor production pipelines and troubleshoot performance, data quality, and availability issues.


• Collaborate with architects and engineering teams to establish data engineering standards and best practices.


• Mentor junior engineers and contribute to technical design and code reviews.

Required Skills & Experience

• 7+ years of overall data engineering experience, with significant hands-on experience in Azure Databricks.


• Strong experience with Apache Spark and PySpark.


• Strong proficiency in SQL and Python.


• Hands-on experience with Delta Lake and lakehouse architecture.


• Experience with Databricks Workflows/Jobs, notebooks, clusters, and job orchestration.


• Experience with Unity Catalog data governance, permissions, and secure data access.


• Strong understanding of Spark performance tuning and optimization.


• Microsoft Azure experience: strong experience with Azure Data Lake Storage Gen2 (ADLS).


• Hands-on experience with Azure Data Factory.


• Experience with Azure Key Vault, Azure Monitor and other Azure data/platform services.


• Good understanding of Azure networking, security, IAM and enterprise cloud architecture.


• Data Engineering: strong understanding of ETL/ELT architecture and data warehousing concepts.


• Experience implementing Medallion Architecture Bronze, Silver and Gold layers.


• Experience with batch and streaming data processing.


• Knowledge of data modelling, dimensional modelling, slowly changing dimensions and CDC.


• Experience working with large-scale datasets and distributed processing.


• Strong understanding of data quality, lineage, observability and governance.


• DevOps & Engineering Practices: experience with Git-based development and CI/CD.


• Experience with Azure DevOps or GitHub Actions.


• Knowledge of automated testing and deployment practices.


• Experience with Infrastructure as Code such as Terraform is desirable.


• Strong understanding of Agile/Scrum delivery methodologies.


• Desirable Skills: Databricks certification such as Databricks Data Engineer Associate/Professional; Microsoft Azure certifications such as Azure Data Engineer Associate (DP-203) or equivalent; Experience with Kafka or Azure Event Hubs; Experience with Power BI and analytical data platforms; Experience with Microsoft Purview or enterprise data governance platforms; Experience with Terraform; Experience with Databricks REST APIs, CLI, SDKs or automation; Experience with machine learning/AI data pipelines is a plus.

Technical Skills

• Azure Databricks | PySpark | Apache Spark | Python | SQL | Delta Lake | ADLS Gen2 | Azure Data Factory | Unity Catalog | Data Engineering | ETL/ELT | Data Modelling | CI/CD | Git


• Good to Have: Azure DevOps | Terraform | Kafka | Azure Event Hubs | Power BI | Microsoft Purview | Databricks Workflows | Structured Streaming | Azure Monitor

Education

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related discipline.

Key Competencies

• Strong analytical and problem-solving skills.


• Excellent communication and stakeholder-management skills.


• Ability to work independently and as part of a distributed engineering team.


• Strong ownership of production data platforms and pipelines.


• Ability to translate business requirements into robust technical solutions.


• Experience working in enterprise environments with high standards for security, quality, and reliability.