Azure Databricks Engineer

Opus Recruitment Solutions Limited; Greater London; Hybrid; GBP 92,000 - 101,000; Full time

Job summary

Opus Recruitment Solutions Limited in London is seeking an experienced Azure Databricks Engineer to design, develop and support scalable data solutions across Azure and Databricks. The role combines hands-on engineering with collaboration across the data team to deliver reliable, high-quality data for business insights.

You will work on pipelines, Delta Lake, Unity Catalog and modern data lake architectures, alongside CI/CD and automation practices.

Qualifications

• Strong hands-on experience with Azure Databricks in an enterprise environment.
• Excellent SQL Server/T-SQL and data engineering skills.
• Experience building and optimising large-scale data pipelines.
• Knowledge of Spark, Delta Lake, Unity Catalog and Azure data services.
• Exposure to Git, CI/CD and infrastructure automation tools.
• Strong communication skills and a proactive approach to problem solving.

Responsibilities

• Develop Databricks solutions using SQL, PySpark and Python.
• Build and maintain ETL/ELT pipelines across Azure, SQL Server and Databricks.
• Support Lakeflow pipelines, production jobs and data quality processes.
• Monitor, troubleshoot and optimise data workflows and platform performance.
• Work with Delta Lake, Unity Catalog and modern data lake architectures.
• Integrate Azure services including ADLS, Data Factory, Key Vault and REST APIs.
• Support CI/CD deployments and Databricks automation.

Skills

Azure Databricks

SQL Server/T-SQL

Data pipelines

Spark

Delta Lake

Unity Catalog

Azure data services

Git

CI/CD

Automation tools

Job description

Azure Databricks Engineer |Inside IR35 | £500 - £550 | Hybrid | London

We're recruiting for an experienced Azure Databricks Engineer to join a growing data team. You'll be responsible for designing, developing and supporting scalable data solutions across Azure and Databricks, helping to deliver reliable, high-quality data for the business. This role would require you to be onsite 4 days per week with the final day being remote.

Key Responsibilities

• Develop Databricks solutions using SQL, PySpark and Python.
• Build and maintain ETL/ELT pipelines across Azure, SQL Server and Databricks.
• Support Lakeflow pipelines, production jobs and data quality processes.
• Monitor, troubleshoot and optimise data workflows and platform performance.
• Work with Delta Lake, Unity Catalog and modern data lake architectures.
• Integrate Azure services including ADLS, Data Factory, Key Vault and REST APIs.
• Support CI/CD deployments and Databricks automation.

Requirements

• Strong hands‑on experience with Azure Databricks in an enterprise environment.
• Excellent SQL Server/T-SQL and data engineering skills.
• Experience building and optimising large-scale data pipelines.
• Knowledge of Spark, Delta Lake, Unity Catalog and Azure data services.
• Exposure to Git, CI/CD and infrastructure automation tools.
• Strong communication skills and a proactive approach to problem solving.

Desirable

• Databricks certifications.
• Experience with Microsoft Fabric, MLflow or Terraform.
• Background working on enterprise data warehouse or data lake projects.

Azure Databricks Engineer |Inside IR35 | £500 - £550 | Hybrid | London