Senior Data Engineer: Azure Databricks & Lakehouse

Opus Recruitment Solutions Limited; Nottingham; On-site; GBP 60,000 - 80,000; Full time

Job summary

Opus Recruitment Solutions Limited is seeking an experienced Data Engineer to design, build, and optimise data platforms in an Azure Databricks environment.

You will develop scalable data pipelines, manage large datasets, and deliver high-quality data solutions within a modern cloud data stack. The role requires hands-on expertise in Python, PySpark, SQL, and Spark SQL, with a focus on orchestration, governance, and performance tuning.

Qualifications

• Strong Python, PySpark and SQL/Spark SQL for building production-grade ETL/ELT pipelines.
• Deep knowledge of Apache Spark: DataFrames, joins, partitions, caching, and performance tuning.
• Experience with Azure Databricks, including clusters, notebooks, jobs and Databricks Runtime; Lakehouse architectures on ADLS Gen2.
• Familiarity with CDC, Auto Loader, batch processing, and modern ingestion patterns; Lakeflow integration and Unity Catalog awareness.

Skills

Python

PySpark

SQL

Spark SQL

Apache Spark

Azure Databricks

Lakehouse

CDC

Git

Azure DevOps

Kafka

Streaming data

Kimball

Job description

Opus Recruitment Solutions Limited is seeking an experienced Data Engineer to design, build, and optimise data platforms in an Azure Databricks environment.

You will develop scalable data pipelines, manage large datasets, and deliver high-quality data solutions within a modern cloud data stack. The role requires hands-on expertise in Python, PySpark, SQL, and Spark SQL, with a focus on orchestration, governance, and performance tuning.