Senior Data Engineer: Petabyte-Scale for Quant Research

Radley James; Greater London; On-site; GBP 180,000 - 220,000; Full time

Job summary

Radley James is partnering with a leading investment firm in London to hire an experienced Data Engineer who can help build and scale a high-performance data platform that supports quantitative investment teams. The role focuses on large-scale data infrastructure, backtesting and research workloads, handling petabyte-scale datasets, and collaborating with quantitative engineers and researchers.

You will design batch and streaming architectures, optimize performance, and improve reliability of

Qualifications

• 5+ years in data-intensive engineering
• Strong SQL and database expertise with large-scale/time-series datasets
• Strong programming in Python, Rust and/or C++
• Experience with Pandas, Polars, Dask or PySpark
• Experience building data platforms, ETL systems, data lakes/warehouses/lakehouse
• Knowledge of Parquet, Arrow or similar formats
• Experience with distributed systems like Kafka and Redis
• Strong understanding of performance optimisation and debugging
• Nice to have: ClickHouse, Snowflake, Prometheus, Grafana or Sentry

Responsibilities

• Building and scaling data infrastructure for backtesting and data-intensive applications
• Developing ingestion and ETL pipelines over petabyte-scale datasets
• Solving data quality, storage, backfills and high-performance data consumption challenges
• Designing batch and streaming data architectures
• Collaborating with quantitative researchers and engineering teams
• Improving scalability, reliability and performance of distributed data systems

Skills

SQL

Python

Rust

C++

Pandas

Polars

Dask

PySpark

Data platforms

ETL pipelines

Parquet/Arrow

Kafka

Redis

Performance optimization

Tools

Kafka

Redis

Job description

Radley James is partnering with a leading investment firm in London to hire an experienced Data Engineer who can help build and scale a high-performance data platform that supports quantitative investment teams. The role focuses on large-scale data infrastructure, backtesting and research workloads, handling petabyte-scale datasets, and collaborating with quantitative engineers and researchers.

You will design batch and streaming architectures, optimize performance, and improve reliability of