Senior Azure Databricks Engineer & Data Pipeline Lead

Capgemini; Greater London; Hybrid; GBP 90,000 - 120,000; Full time

Job summary

Capgemini is seeking a Senior Azure Databricks specialist to design, build and optimise scalable enterprise data platforms within a modern Azure ecosystem. You will design end-to-end data pipelines and data models, ensuring high-quality governed data products across the organisation.

This role operates in a hybrid environment blending Capgemini offices, client sites and remote work as needed. Strong collaboration with cross-functional teams and a focus on data governance and security are

Qualifications

• Hands-on with Azure Databricks and Azure Data Factory.
• Proficient in SQL, Python and PySpark development.
• Experience designing enterprise-grade data pipelines.
• Knowledge of 3NF and dimensional modelling.
• Experience with CI/CD pipelines and DevOps practices.

Responsibilities

• Design, develop and maintain end-to-end data pipelines across Raw, Curated and Data Product layers.
• Build and optimise data ingestion, transformation and processing frameworks using Azure services.
• Implement enterprise data models supporting analytics, reporting and operations.
• Ensure data quality with validation controls and referential integrity checks.
• Drive CI/CD, DevOps practices for data engineering solutions.
• Support data governance including metadata management, data cataloguing and lineage.

Skills

Azure Databricks

Azure Data Factory

SQL

Python

PySpark

Data pipelines

3NF modelling

Dimensional modelling

CI/CD pipelines

DevOps practices

Data governance

Security & compliance

Tools

Azure DevOps

Git

Databricks notebooks

Job description

Capgemini is seeking a Senior Azure Databricks specialist to design, build and optimise scalable enterprise data platforms within a modern Azure ecosystem. You will design end-to-end data pipelines and data models, ensuring high-quality governed data products across the organisation.

This role operates in a hybrid environment blending Capgemini offices, client sites and remote work as needed. Strong collaboration with cross-functional teams and a focus on data governance and security are