Real-Time Data Engineer: Streaming Pipelines & Resilience

Barclays; Glasgow; On-site; GBP 70,000 - 110,000; Full time

Job summary

Barclays in Glasgow seeks a Real-Time Data Engineer to design and build high-volume streaming data pipelines. You will work on ingesting, transforming, and distributing event data with a focus on reliability, observability, and performance.

Successful candidates will have hands-on experience with Python, PySpark, Kafka, and CI/CD processes, plus strong knowledge of distributed data processing and low-latency systems in financial services contexts.

Qualifications

• Strong hands-on experience building data pipelines and streaming solutions using Python, PySpark, Spark Structured Streaming, or similar technologies.
• Solid understanding of distributed data processing concepts, including event ordering, partitioning, consumer groups, stateful processing, and delivery guarantees.
• Experience with real-time data platforms and messaging technologies such as Kafka, Pulsar, Kinesis, Flink, or related streaming frameworks.
• Experience with version control and engineering workflows using Git and GitLab, including support for CI/CD and controlled release processes.
• Experience operating production-grade, low-latency data systems with a focus on resilience, observability, recovery, and data quality.

Responsibilities

• Development and delivery of real-time data pipelines and streaming solutions.
• Collaborate with product managers and engineers to define requirements and ensure reliable, scalable software delivery.
• Ensure secure coding practices, unit testing, and performance optimization across data platforms.
• Promote code quality through reviews and knowledge sharing, staying current with industry trends.

Skills

Python / PySpark

Streaming frameworks

Distributed processing

Kafka / Pulsar / Kinesis

Git / CI/CD

Low-latency data systems

Databricks

Spark SQL

Java / Scala

Financial services domain

Tools

Databricks

GitLab

Job description

Barclays in Glasgow seeks a Real-Time Data Engineer to design and build high-volume streaming data pipelines. You will work on ingesting, transforming, and distributing event data with a focus on reliability, observability, and performance.

Successful candidates will have hands-on experience with Python, PySpark, Kafka, and CI/CD processes, plus strong knowledge of distributed data processing and low-latency systems in financial services contexts.