Middle Databricks Developer with Azure, Data Platform

DataArt; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

DataArt in London is seeking a Strong Middle / Senior Databricks Developer to build scalable data platforms on Microsoft Azure. You will design data solutions with Databricks, develop end-to-end pipelines, and implement Lakehouse architectures using Delta Lake.

You will collaborate with Data Engineers, Data Architects, Analytics teams and business stakeholders to optimise ETL/ELT workflows, ensure data quality, governance and security, and support CI/CD and infrastructure automation.

Qualifications

• 4+ years of experience in Data Engineering.
• 3+ years hands-on experience with Azure Databricks.
• Experience with Python and SQL.
• Experience with end-to-end data pipelines in Azure.
• Understanding of data modeling principles and best practices.
• Experience with large-scale datasets in cloud environments.
• Upper-Intermediate or higher English.

Responsibilities

• Design data solutions using Databricks.
• Develop data pipelines for structured and unstructured data.
• Build ETL and ELT workflows in Azure.
• Develop high performance solutions using PySpark and Spark SQL.
• Design Lakehouse architectures using Delta Lake.
• Integrate data from databases APIs cloud services and streaming platforms.
• Optimize Spark workloads for performance scalability and cost efficiency.
• Ensure data quality governance security and monitoring standards.
• Collaborate with Data Engineers Data Architects Analytics teams and business stakeholders.
• Participate in architectural discussions and technical decision making.
• Support CI and CD processes and infrastructure automation practices.

Skills

Data Engineering

Azure Databricks

Python

SQL

Spark/PySpark

Azure Data Factory

Delta Lake

Git

CI/CD

Cloud Architecture

English (Upper-Intermediate)

Tools

Azure Databricks

Apache Spark

PySpark

Azure Data Factory

Azure Data Lake Storage Gen2

Delta Lake

Spark SQL

Git

Job description

We are looking for a Strong Middle / Senior Databricks Developer with Azure to build scalable data platforms, develop complex data pipelines, and support cloud based data engineering solutions.

The Data Platform project focuses on delivering cloud based data engineering solutions in the Microsoft Azure ecosystem. You will work with Databricks, Spark, and Azure services to build and optimize reliable data pipelines and support modern Lakehouse architecture.

• Design data solutions using Databricks
• Develop data pipelines for structured and unstructured data
• Build ETL and ELT workflows in Azure
• Develop high performance solutions using PySpark and Spark SQL
• Design Lakehouse architectures using Delta Lake
• Integrate data from databases APIs cloud services and streaming platforms
• Optimize Spark workloads for performance scalability and cost efficiency
• Ensure data quality governance security and monitoring standards
• Collaborate with Data Engineers Data Architects Analytics teams and business stakeholders
• Participate in architectural discussions and technical decision making
• Support CI and CD processes and infrastructure automation practices

• 4+ years of experience in Data Engineering
• 3+ years of hands on experience with Azure Databricks
• Experience with Apache Spark and PySpark
• Knowledge of Python and SQL
• Experience with Azure Data Factory
• Experience with Azure Data Lake Storage Gen2
• Experience with Delta Lake and Lakehouse architecture
• Experience implementing end to end data pipelines in Azure
• Understanding of data modeling principles and best practices
• Experience working with large scale datasets in cloud environments
• Experience with Git based version control systems
• Upper Intermediate or higher English level

• Experience optimizing Spark workloads
• Experience collaborating with cross functional technical and business teams
• Experience supporting CI and CD processes
• Experience with infrastructure automation practices