Lead Data Platform Engineer

Carnall Farrar; Greater London; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Holiday entitlement: 25 days/year (30+

Pension: 7% employer contribution

Flexible benefits programme

Annual leave purchase

Income protection

Enhanced sick pay

Life insurance

Diversity & Inclusion programs

Job summary

Carnall Farrar is seeking a Lead Platform Engineer to steward CF’s data platform, translating architectural direction into detailed designs and guiding the data engineering team. You will drive platform standards, mentor engineers and collaborate with data scientists to deliver client-facing data engineering workstreams.

Hybrid role based in the UK, with emphasis on secure, regulated data architectures, multi-region support, and strong vendor relationships across Databricks and NHS data

Qualifications

• Deep expertise in cloud platform design and secure data architecture.
• Experience with Databricks and OCI, or comparable platforms.
• Experience designing for regulated data requirements (NHS IG, ISO 27001, GDPR).
• Track record of setting engineering standards and design patterns.

Responsibilities

• Provide platform architecture and design for CF’s cloud infrastructure and NHS data pipelines.
• Own day-to-day delivery management of platform and data engineering workstreams.
• Define design patterns, reference implementations and ensure quality gates in delivery.
• Collaborate with data scientists and consulting teams to serve client-facing data science delivery.
• Mentor engineers and develop internal learning content on platform architecture.

Skills

Python

SQL

Databricks

OCI

Infrastructure as Code

CI/CD

Git/GitHub

Multi-region

Tools

Cloud platforms

Job description

About us

We are a leading consultancy with a purpose to make an enduring impact on health and healthcare. We work with leaders and frontline teams to improve health, transform healthcare, drive adoption of innovation and create value through investment.

Our consultancy serves the entire healthcare sector, from payors and providers of care, to life science companies, health tech and sector suppliers and health investors. We provide end-to-end services, from strategy through implementation, accelerated by data, digital and AI.

We shape opinion through evidence-based thought leadership on key issues affecting health. With unmatched ability to access and use health data, our consultants are a driving force for delivering positive and meaningful change.

About the role

TheLeadPlatform Engineer is the senior technical lead for CF’s data platform, combining design authority over platform architecture, helping to mentor and directthe data engineering team andmanagingdelivery of platform and data engineering workstreams.

The role sits within CF’s unified data function(Data Innovation and Data Ops), which is led by the Director of Data,Analyticsand Intelligence, asthe most senior role within the function’s Data Opsteam.Within the team this rolesits alongside the Lead Data Scientist(s).

Operatingwith delegated design authority from the Director of Data,Analyticsand Intelligence,the Lead Platform Engineertranslatesthe company’sstrategic architectural direction into detailed technical designs, implementationpatternsand platform standards, enabling the engineering team to execute against them.

Theybring deep technicalexpertise, experience and knowledge tosupport the Director in delivery and the strategic architecture, technology strategy, vendor relationships, commercialtargetsand regulatory frameworks within which the LeadPlatform Engineeroperates.

They help to coach and manage thedata engineer(s)within the DataOperations team. They own the day to day management of the platform and lead data engineering andinfrastructure workstreams.Theyownthe platform, infrastructure and data engineering capability that underpins it. The Lead Platform Engineer works collaboratively with the Lead Data Scientist(s), who manage the data science consulting track anddirect delivery for the client-facing technical consulting team.

TheLeadPlatform Engineer contributes to CF’s commercial activities in a focused way: providing technicalexpertiseto support partnership opportunities with platform vendors such as Databricks and with data and researchorganisationsandcontributing technical input to proposals where platform architecture or infrastructure design is relevant.

Responsibilities

Platform architecture and design

• Produce detailed technical designs for CF’s cloud infrastructure (OCI, Databricks) and NHS data pipelines, working within the strategic architecture set by the Director

• Define integration patterns, data flowdesignsandcomponentspecifications that the data engineering team implements

• Make implementation-level technology decisions (tooling, frameworks, configuration approaches) within the Director’s strategic technology selections

• Design scalable, secure platform architectures that support CF’s multi-region operations (UK, Middle East, EU) andcomply withNHS IG, ISO27001and GDPR requirements

• Own andmaintainplatform design documentation, referencearchitecturesand decision records

• Evaluate emerging platform technologies and make recommendations to the Director on adoption, with supporting technical assessment

Delivery management

• Own day-to-day delivery management of platform and data engineering workstreams in line with the unified function’s strategy and priorities

• Develop andmaintaintechnical workplans for platform initiatives, coordinating with the Director and Lead Data Scientist(s) on sequencing and dependencies

• Manage the data engineering team’s capacity across concurrent workstreams, balancing platform development, Data Operationssupportand client delivery contributions

• Coordinate QA and testing for platform components, ensuring work meets quality standards before release

• Identifyand elevate delivery risks, resourceconstraintsand scope changes to the Director with sufficient lead time

• Ensure platform workstreams are delivered to agreed timelines and within resource constraints

Technical standards and quality

• Define andmaintainengineering standards, designpatternsand reference implementations for the data platform

• Conduct architectural review of platform changes before they enter development, ensuring design coherence and technical soundness

• Contribute to the build quality assurance process for platform components, embedding quality gates in delivery

• Design the security architecture for platform components in line with the Director’s IG framework and ISO 27001 requirements

• Identifyand communicate technical risks and architectural trade-offs, with clear recommendations

Commercial and partnership contribution

• Provide technicalexpertiseto support partnership opportunities with platform vendors such as Databricks, contributing design knowledge and technical credibility to commercial discussions led by the Director and Partners

• Support the Director in technical discussions with data and research organisations where platform architecture or data infrastructure expertise isrequired

• Contribute to the technical sections of proposals and client-facing materials where platform architecture, infrastructure design or data engineering capability is relevant

• Develop an understanding of CF’s growth ambitions and the data platform’s role in enabling them

• Ensure every contact with external stakeholders (technology partners, data organisations, potential recruits) supports the CF brand and aligns with CF values

Collaboration and technical advisory

• Work closely with the Lead Data Scientist(s) to ensure the platform and infrastructure serves client-facing data science delivery

• Advise data scientists, consulting teams and Data Operations colleagues on platform capabilities,constraintsand best-practice usage

• Provide technical input to the Director on infrastructure implications of new data products, clientrequirementsand commercial initiatives

Team apprenticeship, mentorship and capability building

• Mentor and help to scope and support the work of data engineers within the Data Operations team; this includes objective setting, feeding in on performance and development and providing technical day-to-day support

• Act as a technical reference point for the engineering team on platform architecture, cloudinfrastructureand system design

• Develop and deliver internal learning content on platform architecture, designpatternsand infrastructure best practice

• Mentor engineers on architectural thinking, helping the team develop design capability over time

• Support the Director in defining the technical career track and progressionrequirements for data engineering roles

• Contribute to CF’s technical learning programme as faculty where platform and infrastructure topics are relevant

The requirements of the role include:

• Around 4 years of experience in data or platform engineering, preferably including senior or lead responsibility for a production data platform

• Deep expertise in cloud platform design and secure data architecture, including identity and access management, network design and access control; Databricks and OCI, or comparable platforms

• Strong Python and SQL, and experience building and monitoring production pipelines using PySpark or Spark

• Experience with infrastructure as code, CI/CD and a Git and GitHub workflow

• A track record of setting engineering standards and design patterns, conducting architectural review, and producing design documentation that others build from

• Experience owning delivery of technical workstreams: technical workplanning, capacity management, QA and early escalation of risk

• Experience mentoring or managing engineers, including objective setting and development

• Experience designing to regulated data requirements such as NHS IG, ISO 27001 and GDPR, or comparable frameworks

• A clear communicator who can explain design decisions and trade-offs to technical and non-technical audiences

• Comfortable acting as the technical authority for platform and infrastructure decisions, and as the reference point the engineering team comes to

• Confident in commercial and partner-facing technical discussions, including vendor conversations and the technical sections of proposals

Desirable

• Experience with NHS or other routine healthcare data at patient level

• Unity Catalog, Delta Lake and medallion structure, bronze to silver to gold

• Exposure to dbt and dimensional modelling

• Experience contributing to vendor or partner relationships, such as with Databricks

• Experience designing for multi-region operations (UK, Middle East, EU)

Flexible working

We follow a hybrid working model that balances in person connections and remote work to drive exceptional client impact. We enjoy working in person together withclients and colleaguesand work where clients need us to be.

Insupporting flexibility and remote working, team members can work from home one day per week as standard. Additionally, we offer 44 remote working days per year which can be used to top up your working from home days and enable you to work from home up to two days per week-subject to client needs. Alternatively, you could use your allowance in blocks to manage school holidays or other commitments. Our core in person working hoursarefrom 10am until 4pm allowing you that extra flexibility to manage your schedule in a way that works for you.

Our commitment to Diversity & Inclusion

We are committed to building an inclusive and supportive culture where diversity thrives, and all our people can excel. We only recruit, promote and reward our people based on their skills and contribution, without regard to gender, race, disability, religion, nationality, ethnicity, sexual orientation, age, marital status, or other characteristics.

We are Disability Confident Accredited, and we want you to feel comfortable and able to perform at your best in the recruitment process. Ifyourequireany reasonable adjustments for any part of the recruitment process, please let us know.

• Holiday entitlement: 25 days/year for staff and 30 days/year for leadership, increasing by 1 day for every year of service up to a maximum of 35 days of holiday per year

• We contribute 7% of your salary into your pension, while you contribute 3% (or more if you like)

• Access to a flexible benefits programme giving you the chance to increase pension contributions, gain access to a cash plan or benefit from aClassPasssubscription

• Annual leave purchase: employees with less than 35 days annual leave entitlementare able topurchaseadditionalannual leave days

• Income protection:in the event of long-term incapacity and a qualifying claim, 75% of salary will be paid

• Enhanced sick pay benefit beyond Statutory Sick Pay for up to a total 12 weeks in any 12-month period

• Life insurance covering four times your basic salary in a tax-free lump sum payable to your beneficiariesin the event ofyour death whilst in service

• Enhanced family leave policies:additionalpay for parents who have a baby or adopt

• Access to an interest‑free loan of up to £10,000

• Access to an interest‑free season ticket loan, repayable by 12 monthly instalments

• Workplace nursery scheme: access to a scheme to help working parents save tax and NI on the cost of the nursery care

• Flexible working policy: including the ability to work from homeup totwo days per week, with 44 remote working days peryear

• An employeeassistanceand wellness programme: including access to telephone counselling, life coaching, interactive tools online and digital content downloadable from Lifeworks

• Seasonal flu jabs: provided by Boots annually

• Eye care tests: vouchers and discounts at Vision Express

• Ride to work scheme, saving up to 42% on bikes and cycling accessories at Evans Cycles

• Membership to the Health Service Journal (HSJ)