Senior Data Engineer — Azure Databricks Data Platform Lead

CMSPI; Manchester; On-site; GBP 70,000 - 110,000; Full time

Benefits offered by this job

Objective-driven bonuses

Equity opportunities

Professional development

Job summary

CMSPI is seeking a Senior Data Engineer in Manchester to drive the evolution of our payments intelligence platform on Databricks. You will build secure data ingestion, orchestration and analytics pipelines using Databricks as the primary execution layer and Azure services as appropriate.

You will focus on production data engineering, ensuring reliable, observable and governed data foundations that support automated workflows and client onboarding.

Qualifications

• 3+ years professional experience delivering production data platforms.
• Strong hands-on experience with Azure Databricks including Spark/SQL and workflow orchestration.
• Advanced proficiency in Python for data retrieval, processing and automation.
• Experience designing and operating ETL/ELT pipelines with data quality controls.

Responsibilities

• Collaborate within a cross-functional squad to deliver scalable, secure data capabilities aligned with business outcomes.
• Design Databricks-native solutions using Spark/SQL and Databricks assets, with Azure services when needed.
• Build and operate data ingestion across APIs, SFTP/FTPS, migrating orchestration to Databricks-native workflows.
• Own client data onboarding flows including configuration, historical backfills, validation and data quality monitoring.
• Engineer for production reliability with observability, retries and cost optimisation; drive root-cause fixes.
• Deliver changes via CI/CD, automated testing and Infrastructure as Code; maintain secure coding standards.
• Apply security and governance across Unity Catalog, Entra/RBAC and secrets management.
• Use Python and SQL/Spark to develop reusable libraries and data pipelines.
• Mentor engineers and evaluate new Databricks capabilities against business value.

Skills

Azure Databricks

Python

Spark/SQL

APIs & SFTP data integration

CI/CD for data

Entra/RBAC

Data pipelines

Data quality & monitoring

Education

BSc in CS/Math/Physics

Tools

Azure DevOps

Job description

CMSPI is seeking a Senior Data Engineer in Manchester to drive the evolution of our payments intelligence platform on Databricks. You will build secure data ingestion, orchestration and analytics pipelines using Databricks as the primary execution layer and Azure services as appropriate.

You will focus on production data engineering, ensuring reliable, observable and governed data foundations that support automated workflows and client onboarding.