Principal Data Engineer

London Metal Exchange; Greater London; On-site; GBP 120,000 - 180,000; Full time

Job summary

London Metal Exchange is seeking a Senior Data Platform Engineer to shape and deliver a strategic data platform capable of processing billions of events daily. You will work with JVM languages and modern data technologies to build scalable pipelines, data lakes, and real-time streaming components, enabling fast, reliable access to market data for analytics and trading workloads.

The role emphasizes self-service capabilities, platform security, and collaboration across Trading Technology, Data

Qualifications

• 10+ years of software development experience, preferably with JVM-based languages (Java/Scala/Kotlin).
• Expertise in building highly scalable data platforms, data warehouses (SQL Server, Postgres) and modern data lakes.
• Ability to tackle complex scalability, reliability, latency and performance challenges in large-scale distributed systems.
• Experience delivering self-service data platform capabilities for onboarding, processing, discovering and consuming data.
• Influence technology strategy and roadmaps across Trading Tech, Data Eng and Platform Eng functions.

Responsibilities

• Help shape LME's strategic data platform to process, store and serve high-volume, high-velocity trading data.
• Deliver a scalable, resilient data platform serving as the engineering foundation for data-intensive workloads.
• Lead design and implementation of APIs, data products, metadata services and developer enablement frameworks.

Skills

Java/Scala/Kotlin

Distributed systems

Data platform development

Self-service data platforms

Cloud-native platform engineering

Kubernetes/OpenShift

CI/CD / GitOps

Data governance & metadata

Security & regulatory compliance

Technical leadership

Tools

Docker

Kubernetes

Terraform

CI/CD

GitOps

DataHub/OpenMetadata

Apache Kafka

Apache Flink

Apache Spark

Apache Iceberg

S3-compatible storage

Trino

PostgreSQL

SQL Server

Job description

Shift Pattern:

Standard 40 Hour Week (United Kingdom)

Scheduled Weekly Hours:

40

Corporate Grade:

C - Vice President

Reporting Line:

(UK Division) Information Technology

Location:

UK-London

Worker Type:

Permanent

The purpose of this role is to help shape the LME's strategic data platform which can reliably process, store, and serve high-volume, high-velocity trading data at scale, while providing engineering teams with reusable, self-service capabilities that accelerate the delivery of trading and analytical solutions.

Success in this role means delivering a highly scalable and resilient data platform that becomes the engineering foundation for all data-intensive trading workloads, enabling faster innovation, improved reliability, reduced operational complexity, and better utilisation of market and trading data across the organisation.

Strong Required Demonstrable Experience of:

A candidate for this position must have got working experience in IT organisation as a technologist who has evolved from a strong background in hands on software development:

• 10+ years of experience with software development, preferably working with JVM based languages (Java/Scala/Kotlin).
• Provide expertise on the development of highly scalable data platforms, data warehouses (SQL server, Postgres), and modern data lakes.
• Solve complex scalability, reliability, latency, and performance challenges associated with large-scale distributed systems.
• Provide self-service platform capabilities that allow engineering teams to onboard, process, discover, and consume data efficiently.
• Influence technology strategy and roadmaps across Trading Technology, Data Engineering, and Platform Engineering functions.
• Build high-volume distributed data platforms capable of processing billions of events per day and managing terabytes scale datasets.
• Develop modern Data Lakehouse platforms leveraging S3-compatible object storage, Apache Iceberg, Spark, and Trino.
• Build real-time streaming integration using Apache Kafka and Apache Flink for low-latency event processing, enrichment, and data distribution.
• Drive performance engineering and scalability across data ingestion, storage, streaming, query, and analytics platforms.
• Design and build self-service data platform capabilities including APIs, data products, metadata services, and developer enablement frameworks.
• Implement cloud-native platform engineering practices using Kubernetes/OpenShift, Terraform, CI/CD, GitOps, and Infrastructure as Code.
• Lead metadata, lineage, and data discovery capabilities using technologies such as DataHub, OpenMetadata, or Apache Atlas to improve platform transparency and usability.
• Ensure platform security and regulatory compliance through fine-grained access controls, encryption, secrets management, auditing, and secure-by-design engineering principles.

Bonus for knowledge of:

• Automation/configuration management using toolsets such as Puppet, Chef, Ansible or equivalent
• Docker and Kubernetes.
• Knowledge and understanding of financial markets.
• Setting up CI/CD pipelines using Bamboo.
• Confluent certified developer for Apache kafka.
• Dollar Universe

Personal Qualities:

• Ability to work under pressure with changing priorities, with a view to resolving issues innovatively, and meeting key stakeholders expectations.
• A dynamic and self-motivated attitude
• Accountable and proactive
• Able to provide leadership and motivate team demonstrating strong interpersonal skills
• Must display strong analytical skills and attention for detail.
• Demonstrates pragmatic judgement, balancing risk and business value to reach decisions which are well informed and actionable.