Data Engineer

Consortia Group; City Of London; Hybrid; GBP 60,000 - 75,000; Full time

Job summary

A data solutions company in London is seeking a skilled Data Engineer to work with high-volume datasets and cutting-edge technologies. In this role, you will build and maintain data pipelines, integrate real-time data feeds, and support machine learning applications. The ideal candidate has over 3 years of experience in data engineering, strong proficiency in Python and Rust, and hands-on experience with AWS, Linux, and Docker. If you are ready to advance your data engineering career, we want to hear from you!

Qualifications

• 3+ years of experience in data engineering.
• Strong proficiency in Python and Rust.
• Hands-on experience with AWS cloud, Linux, and Docker.
• Expertise in distributed computing frameworks.
• Solid understanding of ETL pipelines and database optimization.

Responsibilities

• Build and maintain high-performance data pipelines.
• Integrate real-time feeds and ensure data flow.
• Implement quality assurance measures for data integrity.
• Support machine learning applications by preparing data.

Skills

Python

Rust

AWS

Linux

Docker

Distributed computing frameworks (Ray, Dask, Apache Arrow)

Data modeling

ETL pipelines

Real-time streaming technologies

Analytical skills

Job description

DataEngineer

London|Hybrid|Full-time

AreyouaskilledDataEngineereagertoworkwithhigh-volumedatasetsandcutting-edgetechnologies?We’relookingforadata-drivenprofessionaltodevelopandoptimiseascalabledataplatformthatsupportsadvancedanalyticsandquantitativeresearch.You’llplayakeyroleindesigningrobustdatapipelines,integratingreal-timefeeds,andensuringtheseamlessflowofstructuredandunstructureddata.

Inthisrole,you'llberesponsibleforbuildingandmaintaininghigh-performancedatapipelinesthatprocesscomplexandlarge-scaledatasets.You'llworkcloselywithanalystsandresearcherstounderstandtheirdatarequirements,optimisingreal-timedataingestionbyintegratingexternalAPIsandalternativedatasources.Youwillfocusonenhancingthedataarchitecturetoimproveaccessibility,efficiency,andsecuritywhileautomatingETLworkflowstostreamlinedatamodellingandanalysis.You’llalsosupportevent-drivensystemsbyimplementingtechnologieslikeKafkaandProtobuf,ensuringseamlessdatastreamingandprocessing.Akeyaspectofyourrolewillbeimplementingrigorousqualityassurancemeasures,ensuringdataintegrityandconsistencyacrossallinputsandoutputs.Additionally,you'llcollaboratewithteamstosupportmachinelearningapplications,facilitatingdatapreparationformodeltraininganddeployment.

WhatYou’llNeed

3+yearsofexperienceinadataengineering

StrongproficiencyinPythonandRust

Hands-onexperiencewithAWScloud,Linux,andDocker.

Expertiseindistributedcomputingframeworks(Ray,Dask,orApacheArrow).

Stronganalyticalskills,withtheabilitytohandlelarge-scale,high-frequencydata

.SolidunderstandingofETLpipelines,datamodelling,anddatabaseoptimisation.

Experienceworkingwithreal-timestreamingtechnologies

Thisisafantasticopportunitytobepartofafast-moving,data-drivenenvironmentwhereyourtechnicalexpertisewillhavearealimpact.You'llworkwithcutting-edgetools,collaboratewithtop-tierprofessionals,andcontributetotheefficiencyandinnovationofagrowingdatateam.

Ifyou’rereadytotakeyourdataengineeringcareertothenextlevel,we’dlovetohearfromyou!