Azure Databricks Engineer

Opus Recruitment Solutions Limited; City Of London; Hybrid; GBP 92,000 - 101,000; Full time

Job summary

Opus Recruitment Solutions Limited is seeking an experienced Azure Databricks Engineer to design, develop, and support scalable data solutions across Azure and Databricks in a hybrid London setting. The role focuses on delivering reliable, high-quality data for the business and requires onsite presence four days per week with one remote day.

You will work on building Databricks solutions, ETL/ELT pipelines, and data quality processes while optimising workflows and performance within Delta Lake

Qualifications

• Strong hands-on experience with Azure Databricks in an enterprise environment.
• Excellent SQL Server/T-SQL and data engineering skills.
• Experience building and optimising large-scale data pipelines.
• Knowledge of Spark, Delta Lake, Unity Catalog and Azure data services.
• Exposure to Git, CI/CD and infrastructure automation tools.
• Strong communication skills and a proactive approach to problem solving.

Responsibilities

• Develop Databricks solutions using SQL, PySpark and Python.
• Build and maintain ETL/ELT pipelines across Azure, SQL Server and Databricks.
• Support Lakeflow pipelines, production jobs and data quality processes.
• Monitor, troubleshoot and optimise data workflows and platform performance.
• Work with Delta Lake, Unity Catalog and modern data lake architectures.
• Integrate Azure services including ADLS, Data Factory, Key Vault and REST APIs.
• Support CI/CD deployments and Databricks automation.

Skills

Azure Databricks

SQL

PySpark

Python

Spark

Delta Lake

Unity Catalog

Azure Data Services

CI/CD

Git

Tools

Git

CI/CD

Job description

Azure Databricks Engineer | Inside IR35 | £500 - £550 | Hybrid | London

We're recruiting for an experienced Azure Databricks Engineer to join a growing data team. You'll be responsible for designing, developing and supporting scalable data solutions across Azure and Databricks, helping to deliver reliable, high-quality data for the business. This role would require you to be onsite 4 days per week with the final day being remote.

Key Responsibilities

• Develop Databricks solutions using SQL, PySpark and Python.
• Build and maintain ETL/ELT pipelines across Azure, SQL Server and Databricks.
• Support Lakeflow pipelines, production jobs and data quality processes.
• Monitor, troubleshoot and optimise data workflows and platform performance.
• Work with Delta Lake, Unity Catalog and modern data lake architectures.
• Integrate Azure services including ADLS, Data Factory, Key Vault and REST APIs.
• Support CI/CD deployments and Databricks automation.

Requirements

• Strong hands-on experience with Azure Databricks in an enterprise environment.
• Excellent SQL Server/T-SQL and data engineering skills.
• Experience building and optimising large-scale data pipelines.
• Knowledge of Spark, Delta Lake, Unity Catalog and Azure data services.
• Exposure to Git, CI/CD and infrastructure automation tools.
• Strong communication skills and a proactive approach to problem solving.

Desirable

• Databricks certifications.
• Experience with Microsoft Fabric, MLflow or Terraform.
• Background working on enterprise data warehouse or data lake projects.

Azure Databricks Engineer | Inside IR35 | £500 - £550 | Hybrid | London