Real-Time Data Engineer: ML Pipelines, Equity, Remote

Vortexa; City Of London; On-site; GBP 60,000 - 80,000; Full time

Job summary

A leading technology firm in London is looking for a skilled professional to design and build backend data processing pipelines. Ideal candidates will have strong software engineering skills and be proficient in both Java and Python. Experience with AWS, Kubernetes, and data lake systems is essential. Join a vibrant and diverse team in a supportive startup environment that values learning and collaboration.

Benefits

Private Health Insurance

Flexible working policy

Global Volunteering Policy

Qualifications

• Experience in building and deploying data processing pipelines with terabytes of data daily.
• Solid software engineering fundamentals.
• Knowledgeable about big data storage formats with a focus on data ingestion.

Responsibilities

• Design and build infrastructure for ML models.
• Ensure 100% uptime and fault-tolerance of data pipelines.
• Collaborate with engineers and analysts to bridge scientific and commercial efforts.

Skills

Building distributed scalable backend data processing pipelines

Fluency in Java and Python

Knowledge of data lake systems

Experience with AWS and Kubernetes (K8s)

Passion for coaching developers

Tools

Apache Kafka

Flink

Airflow

HDF5

Parquet

Job description

A leading technology firm in London is looking for a skilled professional to design and build backend data processing pipelines. Ideal candidates will have strong software engineering skills and be proficient in both Java and Python. Experience with AWS, Kubernetes, and data lake systems is essential. Join a vibrant and diverse team in a supportive startup environment that values learning and collaboration.