Senior Databricks Engineer

Allscreens Nationwide Ltd; Greater London; On-site; GBP 90,000 - 120,000; Part time

Job summary

A senior Databricks Engineer is sought by Allscreens Nationwide Ltd to design and deliver cloud-based data platforms in a UK-based remote/hybrid setting. You will build scalable data pipelines on Azure Databricks, implement Delta Lake architectures, and optimize Spark workloads for enterprise BI and analytics.

The role requires 5+ years of data engineering experience, mastery of Python, PySpark, SQL, and familiarity with Fabric and ADLS Gen2.

Qualifications

• 5+ years of experience in data engineering using Azure technologies.
• Experience delivering cloud-based data platforms for BI and analytics.
• Strong understanding of data warehousing and lakehouse concepts.

Responsibilities

• Design, develop, and maintain enterprise-scale data pipelines using Azure Databricks.
• Build ETL/ELT solutions with Azure Data Factory and Microsoft Fabric.
• Develop data processing with Python, PySpark, and SQL.
• Implement Delta Lake and Lakehouse architectures.
• Work with ADLS Gen2 to manage large-scale datasets.
• Optimize Spark workloads for performance and cost efficiency.
• Design data models to support reporting and analytics.
• Integrate datasets with Power BI for reporting and visualization.
• Implement CI/CD pipelines using Azure DevOps and Git.
• Collaborate with architects, analysts, and stakeholders; troubleshoot production issues.

Skills

Azure Databricks

PySpark

SQL

Python

Delta Lake

Lakehouse architecture

Spark performance tuning

Data modeling

Power BI integration

CI/CD

Tools

Azure Data Factory

Microsoft Fabric

ADLS Gen2

Azure DevOps

Git

Python tooling

Job description

About the Role

We are seeking an experienced Senior Databricks Engineer to join a high-profile client project. This is an excellent opportunity for a skilled data engineering professional with strong expertise in Azure technologies and Microsoft Fabric to design and deliver modern cloud-based data platforms.

Working as part of an experienced delivery team, you will build scalable, high-performance data solutions that support business intelligence, analytics, and enterprise reporting.

Location

Location: UK (Remote/Hybrid – Client Requirement)

Contract Type

Contract Type: Contract

Experience Required

Experience Required: 5+ Years

SC Clearance

SC Clearance: Not Required

Key Responsibilities

• Design, develop, and maintain enterprise-scale data pipelines using Azure Databricks.
• Build robust ETL/ELT solutions using Azure Data Factory and Microsoft Fabric.
• Develop high-performance data processing solutions using Python, PySpark, and SQL.
• Implement Delta Lake and Lakehouse architectures following industry best practices.
• Work with Azure Data Lake Storage (ADLS Gen2) to manage large-scale datasets.
• Optimise Spark workloads for performance, scalability, and cost efficiency.
• Design and implement data models to support reporting and analytics.
• Integrate datasets with Power BI for business reporting and visualisation.
• Implement CI/CD pipelines using Azure DevOps and Git.
• Collaborate with architects, business analysts, and stakeholders to deliver technical solutions.
• Troubleshoot production issues and continuously improve platform performance.

Required Skills & Experience

Why Join Us?

• Opportunity to work on enterprise-scale Azure data projects
• Latest Microsoft Azure and Fabric technologies
• Collaborative and supportive delivery team