Senior Data Engineer

Michael Page; Greater London; On-site; GBP 111,000 - 148,000; Full time

Job summary

Michael Page is seeking a Senior Data Engineer in London to drive customer data outcomes for campaigns and communications. You will own BNA pipeline architecture, data models and the Databricks golden layer, blending engineering, MDM, data architecture and technical leadership.

Role focuses on end-to-end data delivery, data quality, PR governance and production readiness. The contract runs 12 months, rate £600-£800 per day inside IR35, based in Canary Wharf.

Qualifications

• Proficient in SQL, Python/Spark and Databricks data engineering.
• Experience with workflow orchestration and dependency management.
• Familiar with CI/CD, unit/integration testing, and environment promotion.

Responsibilities

• Own BNA pipeline architecture, orchestration & product readiness.
• Own activation data sets, manifests, delivery status & SFTP integration.
• Sequence & approve PRs; act as technical authority before production.
• Enforce end-to-end regression and production validation.
• Own production cutover, run/recovery, observability and incident response.
• Own CRM gold layer facts/dimensions and data movement from source to gold.

Skills

Advanced SQL

Python/Spark

Workflow orchestration

CI/CD

Testing & validation

Tools

Azure DevOps

Databricks

Job description

BNA Senior Data Engineer: sits closer to the business outcome; specifically ensuring customer data is correctly selected & delivered for campaigns / customer communications.

CDI Senior Data Engineer / CRM Golden Layer Principal Data Engineer: owning the enterprise customer data model and golden layer within databricks. Blending engineering, modelling master data management, data architecture & technical leadership.

The organisation operates in the energy industry in London.

Description

• Own BNA pipeline architecture, orchestration & product readiness, implement arbitration, eligibility, suppression, consent, contact policy & cooldown logic. (Designing and maintaining pipelines, ensuring all jobs run in corrrect order and are ready for live use, implementing business rules).
• Own activation data sets / files, manifests, delivery status & SFTP integration - sent to marketing platforms / 3rd parties.
• Sequence & approve PRs (you are the technical authority before code reaches production).
• Enforce end-end regression and production validation
• Own production cutover, run/recovery, observability, alerting and incident technical response.
• Own CRM gold and associated golden layer facts / dimensions (accountable for facts like website activity)
• Design customer, identifier, consent translation, redemption and feedback models (define how customer info is structured).
• Manage source to target mapping, SCD/history handling, lineage & DQ (understanding how customer data moves from source to gold layer)
• Own databricks workflow scheduling, trigger dependencies, freshness & runtime optimisation
• Resolve cross-system data quality & historical reload issuesDefine technical standards, documentation and interface contracts for consumers

Profile

Advanced SQL, Python/Spark and Databricks data engineering.

Workflow orchestration, dependency management, idempotency and scheduled/triggered jobs.

CI/CD, version control Azure DevOps, PR review, unit/integration testing and environment promotion. Production support, DQ controls, alerting, reconciliation and incident diagnosis

Nice to have: CRM decisioning, history with customer eligibility/consent and activation; Practical Comarch and/or SFMC/SFTP interface.

Expert SQL, Python/Spark, Databricks and dimensional/data-product modelling.

SCD1/SCD2, CDC, historical backfill and high-volume transaction processing.

Orchestration, CI/CD, testing, observability and production support.

Nice to have: CRM, loyalty, consent and customer-identity domain work. Ability to work with CIS, Comarch, SFMC, AWS/Data Hub and analytics consumers.

Job Offer

Based in Canary Wharf, Competitive daily rate of £600-£800 a day inside IR35. 12 months contract.