Data Engineer: Scalable Pipelines & Real-Time Analytics

Consortia Group; London; Hybrid; NOK 757,000 - 1,136,000; Full time

Job summary

Consortia Group is seeking a Data Engineer in London to design and optimise scalable data platforms for analytics and research. This hybrid, full-time role focuses on robust data pipelines, real-time ingestion, and data accessibility across teams.

You'll collaborate with analysts and researchers, implement streaming tech like Kafka and Protobuf, and drive data quality, security, and operational excellence in an evolving data-driven environment.

Qualifications

• 3+ years of experience in data engineering.
• Strong proficiency in Python and Rust.
• Hands-on experience with AWS, Linux, and Docker.
• Expertise in distributed computing frameworks (Ray, Dask, or Apache Arrow).
• Strong analytical skills, with the ability to handle large-scale, high-frequency data.
• Solid understanding of ETL pipelines, data modelling, and database optimisation.
• Experience working with real-time streaming technologies.

Responsibilities

• Build and maintain high-performance data pipelines for large-scale datasets.
• Design robust pipelines and real-time ingestion, integrating external APIs and alternative data sources.
• Improve data architecture to enhance accessibility, efficiency, and security.
• Automate ETL workflows to streamline data modelling and analysis.
• Support event-driven systems by implementing technologies like Kafka and Protobuf.
• Ensure data quality and integrity across inputs and outputs.
• Collaborate with analysts and researchers to prepare data for ML models.

Skills

Python

Rust

AWS

Linux

Docker

Ray

Dask

Apache Arrow

Data engineering

Analytics

ETL pipelines

Data modelling

Database optimisation

Real-time streaming

Tools

Kafka

Protobuf

Job description

Consortia Group is seeking a Data Engineer in London to design and optimise scalable data platforms for analytics and research. This hybrid, full-time role focuses on robust data pipelines, real-time ingestion, and data accessibility across teams.

You'll collaborate with analysts and researchers, implement streaming tech like Kafka and Protobuf, and drive data quality, security, and operational excellence in an evolving data-driven environment.