Real-Time Data Engineer: Streaming Pipelines on Azure

Matchtech Group; Greater London; On-site; GBP 85,000 - 110,000; Full time

Benefits offered by this job

Private healthcare

Life assurance (4x salary)

Job summary

Matchtech Group is seeking a Real-Time Data Engineer to design and build high-performance streaming data platforms that process sensor, geospatial and operational data in near real time. You will work on an Azure-based platform, developing scalable data pipelines and reusable platform components for analytics, ML, and decision-making.

You will design, develop and operate streaming solutions across cloud, edge and on-prem environments, focusing on reliability, observability and production

Qualifications

• Strong SQL skills and experience designing analytical data models.

Responsibilities

• Design, build and operate scalable streaming and data processing solutions ingesting, transforming and serving large volumes of real-world data.
• Work across real-time and batch architectures to build reliable, observable, production-ready systems.
• Collaborate with data scientists, software engineers and platform teams to deliver data products for analytics and ML.
• Document data flows, schemas and operational processes.

Skills

SQL

Streaming pipelines

Azure

Docker

Kubernetes

APIs

Time-series data

Data modeling

Distributed processing

Education

Degree in Computer Science or related field

Tools

Apache Flink

Apache Spark

Confluent / Kafka

Job description

Matchtech Group is seeking a Real-Time Data Engineer to design and build high-performance streaming data platforms that process sensor, geospatial and operational data in near real time. You will work on an Azure-based platform, developing scalable data pipelines and reusable platform components for analytics, ML, and decision-making.

You will design, develop and operate streaming solutions across cloud, edge and on-prem environments, focusing on reliability, observability and production