Azure Databricks Architect: Data Pipelines & Governance

Capgemini; Greater London; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

Capgemini seeks a highly experienced Senior Azure Databricks professional to design, build and optimise scalable enterprise data platforms within a modern Azure ecosystem. The role focuses on end-to-end data pipelines, robust data models, and governed data products across the organisation.

Day-to-day work will be a blend of Capgemini offices, client sites, and your home, with limited ability to work from home 100%.

Qualifications

• Strong hands-on with Azure Databricks and Azure Data Factory.
• Proven SQL/Python/PySpark development experience.
• Extensive experience designing enterprise-grade data pipelines.

Responsibilities

• Design, develop and maintain end-to-end data pipelines across Raw, Curated and Data Product layers.
• Build and optimise data ingestion, transformation and processing frameworks using Azure services.
• Develop scalable ETL/ELT solutions leveraging Azure Databricks, Azure Data Factory, SQL, Python and PySpark.
• Implement enterprise data models supporting analytics, reporting and operational use cases.
• Ensure data quality through validation controls, reconciliation and referential integrity checks.
• Design and implement workflow orchestration, monitoring, alerting and SLA management processes.
• Drive performance optimisation across data pipelines, processing jobs and storage layers.
• Contribute to CI/CD and DevOps practices for data engineering solutions.
• Support data governance initiatives including metadata management and access control.

Skills

Azure Databricks

Azure Data Factory

SQL

Python

PySpark

Data Modeling

Data governance

CI/CD

Tools

Azure Databricks

Azure Data Factory

SQL

Job description

Capgemini seeks a highly experienced Senior Azure Databricks professional to design, build and optimise scalable enterprise data platforms within a modern Azure ecosystem. The role focuses on end-to-end data pipelines, robust data models, and governed data products across the organisation.

Day-to-day work will be a blend of Capgemini offices, client sites, and your home, with limited ability to work from home 100%.