Senior Data Engineer

CMSPI; Manchester; On-site; GBP 70,000 - 110,000; Full time

Benefits offered by this job

Objective-driven bonuses

Equity opportunities

Professional development

Job summary

CMSPI is seeking a Senior Data Engineer in Manchester to drive the evolution of our payments intelligence platform on Databricks. You will build secure data ingestion, orchestration and analytics pipelines using Databricks as the primary execution layer and Azure services as appropriate.

You will focus on production data engineering, ensuring reliable, observable and governed data foundations that support automated workflows and client onboarding.

Qualifications

• 3+ years professional experience delivering production data platforms.
• Strong hands-on experience with Azure Databricks including Spark/SQL and workflow orchestration.
• Advanced proficiency in Python for data retrieval, processing and automation.
• Experience designing and operating ETL/ELT pipelines with data quality controls.

Responsibilities

• Collaborate within a cross-functional squad to deliver scalable, secure data capabilities aligned with business outcomes.
• Design Databricks-native solutions using Spark/SQL and Databricks assets, with Azure services when needed.
• Build and operate data ingestion across APIs, SFTP/FTPS, migrating orchestration to Databricks-native workflows.
• Own client data onboarding flows including configuration, historical backfills, validation and data quality monitoring.
• Engineer for production reliability with observability, retries and cost optimisation; drive root-cause fixes.
• Deliver changes via CI/CD, automated testing and Infrastructure as Code; maintain secure coding standards.
• Apply security and governance across Unity Catalog, Entra/RBAC and secrets management.
• Use Python and SQL/Spark to develop reusable libraries and data pipelines.
• Mentor engineers and evaluate new Databricks capabilities against business value.

Skills

Azure Databricks

Python

Spark/SQL

APIs & SFTP data integration

CI/CD for data

Entra/RBAC

Data pipelines

Data quality & monitoring

Education

BSc in CS/Math/Physics

Tools

Azure DevOps

Job description

KEY INFORMATION

Position: Senior Data Engineer

Reporting to: Data Engineering Manager

Location: Manchester

Overview of the Role

As a Senior Data Engineer, you will be an integral part of a cross-functional feature development squad You will play a key role in the evolution of our payments intelligence platform towards a Databricks-native and increasingly agentic operating model. You will design, build and operate secure data ingestion, orchestration and platform capabilities, using Databricks as the first-choice execution platform and Azure services where appropriate. The focus remains on production data engineering rather than AI model development: creating reliable, observable and well-governed foundations that can be operated by both engineers and automated/agentic workflows. You will work across client onboarding, data retrieval, ELT, CI/CD and platform reliability to deliver continuous business value.

JOB ROLE

Key accountability of this role:

Delivery, support and continuous improvement of secure, scalable, Azure/Databricks data capabilities and infrastructure on our payments intelligence platform.

Key responsibilities will include

• Collaborate within a cross-functional squad to define and deliver scalable, secure data and platform capabilities aligned with business outcomes.
• Design Databricks-native solutions as the default, using Spark/SQL, Lakeflow Jobs, Unity Catalog, Volumes and Databricks Asset Bundles, with Azure services used where they add value.
• Build and operate data retrieval and ingestion across APIs, SFTP/FTPS and browser automation, migrating suitable orchestration from external services into Databricks-native workflows.
• Own client data onboarding flows, including configuration, historical backfills, validation, data quality, monitoring and operational handover.
• Build reusable platform capabilities and guardrails that enable agentic workflows to safely automate onboarding, operational diagnosis and routine engineering tasks, with human approval where required.
• Engineer for production reliability through observability, alerting, retries/idempotency, failure recovery, performance and cost optimisation; investigate complex failures and drive root-cause fixes.
• Deliver changes through version control, automated testing, CI/CD and Infrastructure as Code, maintaining secure coding and deployment standards through code review.
• Apply strong security and governance practices across Unity Catalog, Entra/RBAC, secrets, credentials and data access.
• Use advanced Python with strong SQL/Spark skills to develop maintainable, reusable libraries, frameworks and data pipelines.
• Mentor data engineers and drive continuous improvement, evaluating new Databricks and automation capabilities against clear technical and business value.

WHAT WE'RE LOOKING FOR

You are a great match if:

• 3+ years professional experience as a Data Engineer, DevOps Engineer or Systems Engineer delivering production data platforms.
• Strong hands-on experience with Azure Databricks, including Spark/SQL, workflow orchestration and production lakehouse data engineering.
• Advanced proficiency in Python for data retrieval, processing, automation, integration and reusable engineering libraries.
• Experience designing and operating ETL/ELT pipelines with data quality controls, schema handling, monitoring and failure recovery.
• Expertise in CI/CD, version-controlled deployment and automated testing for data or platform workloads.
• Experience integrating data through APIs and secure file transfer (SFTP/FTPS), including secure handling of credentials and secrets.
• Strong Microsoft Azure knowledge, including ADLS, Azure DevOps and Entra/RBAC, with experience designing and deploying infrastructure.
• Strong production troubleshooting, problem-solving and operational support skills.
• Degree in Computer Science, Mathematics, Physics or other STEM related subject.

What we offer…

• Excellent performance-based earning opportunity, including Objective-driven bonuses.
• Ability to advance career and expand professional experiences in a hyper-growth company.
• Future opportunity for equity, rewarded to high performers.
• Payments industry training and continuous training in respective role.
• Personalised individual development plan, aligned to professional goals.