Senior Data Engineer - Global Algorithmic Trading

Darwin Partners; Brussel; Sur place; EUR 70 000 - 100 000; Plein temps

Résumé du poste

Darwin Partners is building the next generation data platform for real-time algorithmic trading. You will design, deploy and operate data infrastructure powering trading models, signals and market data across multiple markets.

You will collaborate with AI engineers, quantitative developers and traders to ensure data quality, traceability and performance. The role offers an opportunity to industrialise DataOps and MLOps practices in production.

Qualifications

• Five to seven years in Data Engineering or related field.
• Advanced Python skills and strong software fundamentals.
• Experience with real-time data pipelines and production-grade systems.
• Proficient with Kafka, RabbitMQ or similar messaging systems.
• Hands-on with Docker, Kubernetes and cloud environments.

Responsabilités

• Design, build and optimise low-latency data architectures.
• Define foundations to support algorithmic trading and large-scale data processing.
• Select storage, streaming and distributed processing technologies.
• Contribute to architecture standards and best practices.
• Develop, automate and monitor ETL/ELT pipelines.
• Ingest real-time data: prices, trades, positions, market data, signals.
• Ensure scalability, resilience and performance of data pipelines.
• Implement event replay, backfilling, recovery and reconciliation.

Connaissances

Python

C++

Rust

Java

Scala

Software engineering fundamentals

Real-time streaming

Distributed systems

Team collaboration

Production mindset

Outils

Apache Kafka

RabbitMQ

Docker

Kubernetes

TimescaleDB

InfluxDB

ClickHouse

KDB+

AWS

Azure

GCP

Description du poste

We are supporting a major international trading organisation in the continued development of its algorithmic trading platform.

The team designs advanced trading algorithms and integrates Artificial Intelligence and quantitative models to analyse market conditions, generate signals and support real-time trading decisions across multiple markets.

Your role

You will design, deploy and maintain the data infrastructure required by real-time and near-real-time trading algorithms.

You will work closely with three core groups:

• Algorithm and quantitative development teams;
• Software engineers responsible for building and deploying services;
• Research, trading and business stakeholders.

Your objective will be to ensure that market data, pricing information, forecasts, trading signals and model outputs are available, reliable, traceable and processed with the appropriate level of performance.

Key responsibilities

• Design, build and optimise robust, scalable and low-latency data architectures.
• Define the technical foundations required to support algorithmic trading and large-scale market data processing.
• Select and implement the most appropriate storage, streaming and distributed processing technologies.
• Contribute to the definition of architecture standards and engineering best practices across the trading platform.
• Develop, automate and monitor complex ETL and ELT pipelines.
• Ingest and process real-time and near-real-time data, including:
• market prices and order book data;
• transactions, positions and execution data;
• external market and reference data;
• pricing, forecasting and optimisation inputs;
• signals and outputs generated by quantitative and AI models.
• Ensure the scalability, resilience and performance of the pipelines used by the trading algorithms.
• Implement mechanisms for event replay, backfilling, recovery and reconciliation.

DataOps and MLOps industrialisation

• Work closely with AI Engineers and Quantitative Developers to structure the data workflows used by trading models.
• Support the transition from research prototypes to scalable and reliable production services.
• Build reusable data components and standardise development, testing and deployment practices.
• Contribute to feature pipelines, model data preparation and production monitoring.
• Reduce the time required to move new algorithms and models from research into production.

Data quality, tracking and governance

• Implement automated data quality controls and validation mechanisms.
• Ensure data availability, integrity, consistency and traceability.
• Introduce appropriate tracking, lineage and governance practices.
• Define monitoring, alerting and recovery mechanisms for critical data flows.
• Manage challenges such as late events, duplicated messages, missing data, schema evolution, replay and backfilling.
• Establish clear indicators covering data freshness, pipeline performance and platform availability.

Production and collaboration

• Deploy and operate data services in production environments.
• Contribute to CI/CD, Infrastructure as Code and automated deployment practices.
• Investigate production incidents and improve platform observability and reliability.
• Work closely with Software Engineers, Quantitative Developers, Researchers and Traders.
• Act as the technical interface between Technology, Quantitative Research and Trading teams.
• Promote a strong production and engineering culture across the desk.

Technical environment

The platform relies on a modern real-time data and cloud stack, including:

• Programming: Python, with at least one additional performance-oriented language such as C++, Rust, Java or Scala;
• Streaming and messaging: Apache Kafka, RabbitMQ or equivalent event-driven technologies;
• Databases: time-series and analytical databases such as TimescaleDB, InfluxDB, ClickHouse or KDB+;
• Distributed processing: Apache Spark, Apache Flink or similar distributed computing frameworks;
• Containers and orchestration: Docker and Kubernetes;
• Cloud: AWS, Microsoft Azure or Google Cloud Platform;
• Industrialisation: CI/CD, Infrastructure as Code, DataOps and MLOps practices;
• Observability: monitoring, logging, alerting, tracing and data lineage tools.

The technology stack continues to evolve, and you will have the opportunity to influence architecture decisions, technology choices and engineering standards.

Your background

• At least five to seven years of experience in Data Engineering, Data Platform Engineering, Backend Engineering or a related position.
• Advanced Python skills and strong software engineering fundamentals.
• Professional experience with at least one additional language such as C++, Rust, Java or Scala.
• Strong experience designing, building and operating production-grade data pipelines.
• Hands-on experience with real-time streaming and event-driven architectures.
• Experience with Apache Kafka, RabbitMQ or comparable technologies.
• Good knowledge of time-series, analytical, SQL or NoSQL databases.
• Strong experience with Docker, Kubernetes and cloud environments.
• Good understanding of distributed systems, scalability, performance and fault tolerance.
• Strong production mindset, including testing, observability, incident management and continuous improvement.
• Ability to collaborate effectively with both technical and business-oriented stakeholders.