Data Engineer- BPL-CIO

Barclays; City Of London; Hybrid; GBP 80,000 - 110,000; Full time

Job summary

Barclays is seeking a Data Engineer in London to build and maintain data pipelines, data warehouses and data lakes with a security-first mindset. You will collaborate with data scientists to deploy machine learning models and design scalable data platforms using Databricks on AWS, adopting IaC and CI/CD practices.

You will work in a hybrid setup with office presence in London, contributing to a high-scale payments data platform and guiding domain teams with reusable, self-service data products.

Qualifications

• Hands-on with production-grade Databricks platforms and batch/streaming pipelines (Python, PySpark, SQL).
• CI/CD pipelines and automated testing using GitHub Actions or similar.
• Security-focused data engineering including IAM, encryption, secrets management, least privilege.
• Unity Catalog configuration for lineage, permissions, metadata and governance.
• IaC with Terraform and Databricks deployment automation.
• Platform product mindset enabling self-service data products.

Responsibilities

• Build and maintain data pipelines, ensuring durable, complete, and consistent data.
• Design data warehouses and data lakes with appropriate scale and security.
• Develop processing and analysis algorithms for complex data.
• Collaborate with data scientists to deploy machine learning models.

Skills

Databricks

Python

PySpark

SQL

CI/CD

Terraform

GitHub Actions

Identity & Access Control

Encryption

Unity Catalog

Tools

Databricks

Terraform

GitHub Actions

Unity Catalog

Job description

Job Description

Purpose of the role

To build and maintain the systems that collect, store, process, and analyse data, such as data pipelines, data warehouses and data lakes to ensure that all data is accurate, accessible, and secure.

Accountabilities

• Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data.
• Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures.
• Development of processing and analysis algorithms fit for the intended data complexity and volumes.
• Collaboration with data scientist to build and deploy machine learning models.

Vice President Expectations

• To contribute or set strategy, drive requirements and make recommendations for change. Plan resources, budgets, and policies; manage and maintain policies/ processes; deliver continuous improvements and elevate breaches of policies/procedures..
• If managing a team, they define jobs and responsibilities, planning for the department's future needs and operations, counselling employees on performance and contributing to employee pay decisions/changes. They may also lead a number of specialists to influence the operations of a department, in alignment with strategic as well as tactical priorities, while balancing short and long term goals and ensuring that budgets and schedules meet corporate requirements..
• If the position has leadership responsibilities, People Leaders are expected to demonstrate a clear set of leadership behaviours to create an environment for colleagues to thrive and deliver to a consistently excellent standard. The four LEAD behaviours are: L - Listen and be authentic, E - Energise and inspire, A - Align across the enterprise, D - Develop others..
• OR for an individual contributor, they will be a subject matter expert within own discipline and will guide technical direction. They will lead collaborative, multi-year assignments and guide team members through structured assignments, identify the need for the inclusion of other areas of specialisation to complete assignments. They will train, guide and coach less experienced specialists and provide information affecting long term profits, organisational risks and strategic decisions..
• Advise key stakeholders, including functional leadership teams and senior management on functional and cross functional areas of impact and alignment.
• Manage and mitigate risks through assessment, in support of the control and governance agenda.
• Demonstrate leadership and accountability for managing risk and strengthening controls in relation to the work your team does.
• Demonstrate comprehensive understanding of the organisation functions to contribute to achieving the goals of the business.
• Collaborate with other areas of work, for business aligned support areas to keep up to speed with business activity and the business strategies.
• Create solutions based on sophisticated analytical thought comparing and selecting complex alternatives. In-depth analysis with interpretative thinking will be required to define problems and develop innovative solutions.
• Adopt and include the outcomes of extensive research in problem solving processes.
• Seek out, build and maintain trusting relationships and partnerships with internal and external stakeholders in order to accomplish key business objectives, using influencing and negotiating skills to achieve outcomes.

All colleagues will be expected to demonstrate the Barclays Values of Respect, Integrity, Service, Excellence and Stewardship - our moral compass, helping us do what we believe is right. They will also be expected to demonstrate the Barclays Mindset - to Empower, Challenge and Drive - the operating manual for how we behave.

Join us as a Data Engineer and help build the data platform that will underpin a new standalone payments business.

As one of the first permanent engineers in the team, you will play a hands-on role in both shaping the platform and delivering the initial data products that run on it. Working with Databricks and AWS, you will design and build production-grade data pipelines, establish engineering standards and create the reusable patterns, automation and controls that will form the foundation of the platform.

This is not a role where platform engineering is separated from delivery. We believe the best platform capabilities emerge from solving real business problems. You will work closely with domain teams, building early data products, identifying common requirements and turning successful approaches into repeatable golden paths that make it easier for future teams to onboard and succeed.

As the platform evolves, your focus will increasingly shift from hands-on support towards self-service capabilities, reusable tooling and platform features that enable domain teams to build and operate trusted data products independently.

To be successful as a Data Engineer, you must have the following essential skills;

Production Databricks and data pipeline engineering

• Hands-on experience building production-grade Databricks platforms and reliable batch and streaming pipelines using Python, PySpark and SQL.

Software engineering, testing and automated delivery

• The ability to create modular, maintainable and reusable components, supported by automated pipeline testing, Git-based workflows and CI/CD deployment.

Secure and operable platform engineering

• Experience engineering systems for sensitive or regulated data, including identity and access control, encryption, secrets management, least privilege, monitoring and production incident resolution.

Advanced Unity Catalog expertise

• Practical experience configuring lineage, permissions, metadata and governance through Unity Catalog.

Infrastructure-as-code and Databricks deployment automation

• Experience with Terraform, Databricks Asset Bundles and GitHub Actions or equivalent policy-controlled delivery tooling.

Platform product and self-service engineering

• Experience creating paved roads, golden paths, metadata-driven automation or internal developer platforms that enable domain teams to build compliant data products with less manual support.

Some other highly valued skills may include;

The successful candidate will be based in London. Our offices are located at 1 Churchill Place and 7 Westferry Circus (new BPL office). We support a hybrid working pattern with 3 days per week office-based presence expected.

Barclays' payments acceptance business provides critical infrastructure to the UK economy, processing billions of pounds of payments annually for both small businesses and domestic and international corporate clients.

In April 2025, we announced a long-term partnership with Brookfield Asset Management to grow and transform the payments acceptance business by broadening the range of services offered, enhancing the experience for both existing and prospective clients. Leveraging extensive client relationships and deep experience of UK payments, we will create an environment of continuous innovation - activated by Brookfield's global private equity expertise in payments, technology, operational transformation and corporate carve-outs - to ensure the business is strategically positioned for long-term growth.

Barclays will invest approximately £400m in the new business, the majority of which will be incurred during the first three years. Performance-linked incentives will drive greater alignment between the partners, underpinning the long-term commitment to the transformation. Barclays and Brookfield will work to create a standalone entity over time, continuing to use the Barclaycard Payments (BPL) brand and acting as the sole payments' acceptance services provider to Barclays' clients for a minimum of ten years.

For more information on our partnership with Brookfield, please visit Barclays.com.