Senior Databricks Engineer - Azure Lakehouse & Spark

DataArt; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

DataArt in London is seeking a Strong Middle / Senior Databricks Developer to build scalable data platforms on Microsoft Azure. You will design data solutions with Databricks, develop end-to-end pipelines, and implement Lakehouse architectures using Delta Lake.

You will collaborate with Data Engineers, Data Architects, Analytics teams and business stakeholders to optimise ETL/ELT workflows, ensure data quality, governance and security, and support CI/CD and infrastructure automation.

Qualifications

• 4+ years of experience in Data Engineering.
• 3+ years hands-on experience with Azure Databricks.
• Experience with Python and SQL.
• Experience with end-to-end data pipelines in Azure.
• Understanding of data modeling principles and best practices.
• Experience with large-scale datasets in cloud environments.
• Upper-Intermediate or higher English.

Responsibilities

• Design data solutions using Databricks.
• Develop data pipelines for structured and unstructured data.
• Build ETL and ELT workflows in Azure.
• Develop high performance solutions using PySpark and Spark SQL.
• Design Lakehouse architectures using Delta Lake.
• Integrate data from databases APIs cloud services and streaming platforms.
• Optimize Spark workloads for performance scalability and cost efficiency.
• Ensure data quality governance security and monitoring standards.
• Collaborate with Data Engineers Data Architects Analytics teams and business stakeholders.
• Participate in architectural discussions and technical decision making.
• Support CI and CD processes and infrastructure automation practices.

Skills

Data Engineering

Azure Databricks

Python

SQL

Spark/PySpark

Azure Data Factory

Delta Lake

Git

CI/CD

Cloud Architecture

English (Upper-Intermediate)

Tools

Azure Databricks

Apache Spark

PySpark

Azure Data Factory

Azure Data Lake Storage Gen2

Delta Lake

Spark SQL

Git

Job description

DataArt in London is seeking a Strong Middle / Senior Databricks Developer to build scalable data platforms on Microsoft Azure. You will design data solutions with Databricks, develop end-to-end pipelines, and implement Lakehouse architectures using Delta Lake.

You will collaborate with Data Engineers, Data Architects, Analytics teams and business stakeholders to optimise ETL/ELT workflows, ensure data quality, governance and security, and support CI/CD and infrastructure automation.