Senior Azure Databricks Data Platform Lead

korecv; Greater London; Hybrid; GBP 90,000 - 140,000; Full time

Benefits offered by this job

Discretionary bonus

Competitive pension

Health insurance

Life insurance

Job summary

Capco seeks a Principal Azure Data Engineer (Databricks) to architect and lead Azure-based data platforms. You will design scalable, secure pipelines with Delta Lake, Unity Catalog, and Spark Streaming, while guiding clients on data strategy and governance.

You will mentor teams, promote responsible AI tools, and ensure quality, security, and compliance across deployments in a hybrid London environment.

Qualifications

• Expert in Databricks on Azure with Unity Catalog and Delta Lake.
• Strong Python/PySpark skills and streaming data design.
• Experience building production-grade data platforms and CI/CD pipelines.
• Knowledge of GDPR-compliant data solutions and governance.
• Familiarity with AI-assisted engineering practices.

Responsibilities

• Architect and lead end-to-end Databricks-based data platforms on Azure.
• Design secure, scalable data pipelines using Delta Lake and Spark Streaming.
• Establish governance and quality standards across deployments.
• Partner with clients to define data strategy and governance.
• Mentor teams and promote responsible AI tooling.

Skills

Databricks

Unity Catalog

Delta Lake

Spark Structured Streaming

Python

PySpark

Azure DevOps

Jenkins

GitHub Actions

Data Lakehouse

GDPR compliance

AI tools

Tools

Azure DevOps

Jenkins

GitHub Actions

Job description

Capco seeks a Principal Azure Data Engineer (Databricks) to architect and lead Azure-based data platforms. You will design scalable, secure pipelines with Delta Lake, Unity Catalog, and Spark Streaming, while guiding clients on data strategy and governance.

You will mentor teams, promote responsible AI tools, and ensure quality, security, and compliance across deployments in a hybrid London environment.