Azure Databricks Lead Engineer

Capgemini; Greater London; Hybrid; GBP 65,000 - 95,000; Full time

Job summary

Capgemini is seeking an Azure Databricks Engineer to join the ACT Practice in the UK. The role focuses on hands-on Databricks implementation, pipeline design, and integration across cloud platforms.

Working in a hybrid model, you will collaborate with Data Engineers and Platform Teams to deliver scalable data solutions for clients. You will engage in cloud identity, security, monitoring, and cost optimisation while supporting modernization initiatives and migration activities.

Qualifications

• Hands-on implementation and troubleshooting of Azure Databricks and Serverless configuration
• Python, PySpark, and SQL development for data pipelines
• REST API integration and secure credential management
• Azure Identity, Networking, Security, and Monitoring
• Cost monitoring, tagging, and budget control practices
• Production support, incident management, and governance

Responsibilities

• Hands-on implementation and troubleshooting of Azure Databricks
• Databricks Serverless configuration and workload optimisation
• Databricks SQL and Delta Lake development
• Cluster management, compute policies, autoscaling, and workload monitoring
• Strong Python, PySpark, and SQL development skills
• Design and build ingestion and transformation pipelines
• Full and incremental data loading
• CDC implementation
• Data quality controls, schema evolution, reconciliation, and error handling
• REST API integration and secure authentication

Skills

Azure Databricks

Data Engineering

Integration Development

Azure Platform Services

FinOps

Platform Operations

Migration Support

Collaboration

Job description

Capgemini is seeking an Azure Databricks Engineer to join the ACT Practice in the UK. The role focuses on hands-on Databricks implementation, pipeline design, and integration across cloud platforms.

Working in a hybrid model, you will collaborate with Data Engineers and Platform Teams to deliver scalable data solutions for clients. You will engage in cloud identity, security, monitoring, and cost optimisation while supporting modernization initiatives and migration activities.