Senior Data Engineer, Real-Time Commodity Analytics on AWS

Kpler; United Kingdom; On-site; GBP 90,000 - 140,000; Full time

Job summary

Kpler in the United Kingdom is seeking a Senior Data Engineer for the Financial Flows team. You will own end-to-end data pipelines, predictive models, and cloud infrastructure powering commodity finance analytics, migrating services to AWS and rewriting legacy components in Python for performance and reliability.

Collaborate with quantitative researchers, analysts, and product managers to translate trading models into high-performance production pipelines, and mentor peers to raise engineering

Qualifications

• 5+ years of professional experience building and scaling production-grade data pipelines.
• Strong Python expertise with production-quality code refactoring.
• Experience with real-time or streaming ingestion pipelines (REST, WebSockets).
• Experience deploying containerized apps on AWS with Docker and Kubernetes.
• Strong SQL fundamentals and experience with distributed time-series databases.
• Automated data quality, observability, and root-cause troubleshooting in production.

Responsibilities

• Design, build, and operate batch and real-time data ingestion pipelines via APIs and WebSockets.
• Lead migration of legacy backend to AWS cloud-native environments with high reliability.
• Refactor CTA positioning indicator calculations from PHP to Python with parity and tests.
• Ingest and model futures pricing, open interest, and volumes across time-series stores.
• Implement automated validation, alerting, and monitoring for data feeds.
• Deploy, maintain, and scale containers with CI/CD and GitOps.
• Collaborate with researchers, analysts, and product managers to productionize trading models.
• Provide mentorship through code reviews, design guidance, and documentation.

Skills

Python

Data pipelines

Cloud (AWS)

CI/CD

SQL

Mentorship

Tools

AWS

Docker

Kubernetes

Elasticsearch

QuestDB

ClickHouse

GitOps

Job description

Kpler in the United Kingdom is seeking a Senior Data Engineer for the Financial Flows team. You will own end-to-end data pipelines, predictive models, and cloud infrastructure powering commodity finance analytics, migrating services to AWS and rewriting legacy components in Python for performance and reliability.

Collaborate with quantitative researchers, analysts, and product managers to translate trading models into high-performance production pipelines, and mentor peers to raise engineering