Senior Azure Data Engineer (Databricks) - Real-Time Pipelines

Capco

City of Edinburgh

Hybrid

GBP 70,000 - 100,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Discretionary bonus
Health insurance
Pension
Life insurance
Mental health support
Family leave

Job summary

Capco seeks a Senior Azure Data Engineer (Databricks) to design, build, and deploy scalable data pipelines on Azure in a world-class consulting environment. You’ll partner with clients to translate data needs into robust solutions and contribute to best practice engineering across cross-functional teams.

Based in Scotland with hybrid work from Edinburgh or Glasgow, you’ll work with DeltaLake, Spark, Kafka, and Unity Catalog, applying DevOps ideas to CI/CD pipelines and GDPR-compliant data

Qualifications

  • Proven hands-on experience with Databricks platform and orchestration.
  • Strong skills in Python, PySpark, and SQL, with knowledge of distributed data systems.
  • Expertise in developing full lifecycle data pipelines across ingestion, transformation, and serving layers.
  • Experience with data lakehouse architecture, schema design, and GDPR-compliant solutions.
  • Working knowledge of DevOps tools and CI/CD processes.

Responsibilities

  • Design and develop robust pipelines using DeltaLake, Spark Structured Streaming, and Unity Catalog.
  • Build real-time event-driven solutions with tools such as Kafka and Azure Event Hubs.
  • Apply DevOps principles to develop CI/CD pipelines using Azure DevOps, Jenkins, or GitHub Actions.
  • Collaborate with clients and stakeholders to translate data needs into strategic technical solutions.
  • Champion clean code, data lifecycle optimisation, and software engineering best practices.

Skills

Databricks
Python
PySpark
SQL
Delta Lake
Spark
Kafka
Azure DevOps
CI/CD
GDPR
Data Pipelines

Tools

Azure
GitHub Actions
Jenkins
Unity Catalog

Job description

Capco seeks a Senior Azure Data Engineer (Databricks) to design, build, and deploy scalable data pipelines on Azure in a world-class consulting environment. You’ll partner with clients to translate data needs into robust solutions and contribute to best practice engineering across cross-functional teams.

Based in Scotland with hybrid work from Edinburgh or Glasgow, you’ll work with DeltaLake, Spark, Kafka, and Unity Catalog, applying DevOps ideas to CI/CD pipelines and GDPR-compliant data

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Azure Data Engineer (Databricks) – Real-Time Pipelines
Senior Azure Data Engineer (Databricks) – Real-Time Pipelines

Capco • Glasgow

Hybrid
GBP 90,000 - 120,000
Discretionary bonus
Pension
Health insurance
+3
Azure Databricks Engineer for Data Pipelines & Lakes
Azure Databricks Engineer for Data Pipelines & Lakes

Capgemini • Greater London

Hybrid
GBP 70,000 - 100,000
Disability Confident Employer (Level 2
Data Engineer - Azure Databricks Data Pipelines
Data Engineer - Azure Databricks Data Pipelines

SSE Enterprise • Reading

Hybrid
GBP 36,000 - 54,000
Private healthcare discounts
Gym memberships
Online GP service
+3
Databricks Data Engineer: Azure Data Pipelines & AI-Ready
Databricks Data Engineer: Azure Data Pipelines & AI-Ready

Dufrain Consulting Ltd • England

On-site
GBP 70,000 - 100,000
Senior Azure Data Engineer (Databricks)
Senior Azure Data Engineer (Databricks)

Capco • City of Edinburgh

Hybrid
GBP 70,000 - 100,000
Discretionary bonus
Health insurance
Pension
+3
Senior Azure Data Engineer (Databricks)
Senior Azure Data Engineer (Databricks)

Capco • Glasgow

Hybrid
GBP 90,000 - 120,000
Discretionary bonus
Pension
Health insurance
+3
Senior Azure Data Engineer — Databricks, ADF, Python
Senior Azure Data Engineer — Databricks, ADF, Python

CGI Group Inc. • City of Westminster

Hybrid
GBP 65,000 - 90,000
Insurance coverage
Medical benefits
Pension plan
+3
Azure Databricks Lead Engineer | Data & Platform Ops
Azure Databricks Lead Engineer | Data & Platform Ops

Sogeti • United Kingdom

Hybrid
GBP 70,000 - 110,000
Data Engineer: Azure Databricks & Cloud Pipelines (Hybrid)
Data Engineer: Azure Databricks & Cloud Pipelines (Hybrid)

Standard Life plc • City of Edinburgh

Hybrid
GBP 35,000 - 45,000
Bonus
Private medical cover
38 days annual leave
+5
Lead Azure Databricks Architect for Real-Time Data (Hybrid)
Lead Azure Databricks Architect for Real-Time Data (Hybrid)

EPAM Systems Inc • City of Westminster

Hybrid
GBP 120,000 - 150,000
ESPP
Private medical insurance
Dental care
+4