Principal Data Engineer

JobCubby; Greater London; Hybrid; GBP 120,000 - 180,000; Full time

Job summary

London Metal Exchange (LME) is seeking a Principal Data Engineer in London to design and build a highly scalable data platform for high-volume trading data.

The role focuses on real-time streaming, data lakehouse architectures, and cloud-native infrastructure, with leadership across data and platform engineering teams.

Qualifications

• Requires 10+ years of software development experience, with JVM languages (Java/Scala/Kotlin).
• Experience building highly scalable data platforms and data lakes/data warehouses.
• Proven ability to tackle latency, scalability, and reliability challenges in large distributed systems.
• Experience delivering self-service data platform capabilities for engineering teams.

Responsibilities

• Design and build a scalable data platform to support high-volume trading workloads.
• Provide self-service capabilities for ingestion, discovery, and consumption of data.
• Influence technology strategy and roadmaps across data, platform and trading tech.
• Lead data ingestion, storage, streaming, queries, and analytics platforms.

Skills

JVM languages

Distributed systems

Data lakehouse

Real-time streaming

Cloud-native

Self-service platforms

Leadership

Performance/systems tuning

Tools

Apache Kafka

Apache Flink

Apache Spark

Apache Iceberg

Trino

Kubernetes/OpenShift

Terraform

CI/CD

DataHub/OpenMetadata

Apache Atlas

Job description

to apply - email only, no card. You can also save this posting or score it againstyour profile with AI.## About the roleThe Principal Data Engineer will design and build a highly scalable, resilient data platform to support high-volume trading workloads. They will also provide self-service capabilities and influence technology strategy across the organization's data and platform engineering functions.## RequirementsCandidates must have over 10 years of software development experience, specifically with JVM-based languages and large-scale distributed systems. Expertise in modern data lakehouse architectures, real-time streaming, and cloud-native infrastructure is essential for this role.## Full descriptionPrincipal Data EngineerShift Pattern:Standard 40 Hour Week (United Kingdom)Scheduled Weekly Hours:40Corporate Grade:C - Vice PresidentReporting Line:(UK Division) Information TechnologyLocation:UK-LondonWorker Type:PermanentThe purpose of this role is to help shape the LME's strategic data platform which can reliably process, store, and serve high-volume, high-velocity trading data at scale, while providing engineering teams with reusable, self-service capabilities that accelerate the delivery of trading and analytical solutions.Success in this role means delivering a highly scalable and resilient data platform that becomes the engineering foundation for all data-intensive trading workloads, enabling faster innovation, improved reliability, reduced operational complexity, and better utilisation of market and trading data across the organisation.Strong Required Demonstrable Experience of:A candidate for this position must have got working experience in IT organisation as a technologist who has evolved from a strong background in hands on software development:* 10+ years of experience with software development, preferably working with JVM based languages (Java/Scala/Kotlin).* Provide expertise on the development of highly scalable data platforms, data warehouses (SQL server, Postgres), and modern data lakes.* Solve complex scalability, reliability, latency, and performance challenges associated with large-scale distributed systems.* Provide self-service platform capabilities that allow engineering teams to onboard, process, discover, and consume data efficiently.* Influence technology strategy and roadmaps across Trading Technology, Data Engineering, and Platform Engineering functions.* Build high-volume distributed data platforms capable of processing billions of events per day and managing terabytes scale datasets.* Develop modern Data Lakehouse platforms leveraging S3-compatible object storage, Apache Iceberg, Spark, and Trino.* Build real-time streaming integration using Apache Kafka and Apache Flink for low-latency event processing, enrichment, and data distribution.* Drive performance engineering and scalability across data ingestion, storage, streaming, query, and analytics platforms.* Design and build self-service data platform capabilities including APIs, data products, metadata services, and developer enablement frameworks.* Implement cloud-native platform engineering practices using Kubernetes/OpenShift, Terraform, CI/CD, GitOps, and Infrastructure as Code.* Lead metadata, lineage, and data discovery capabilities using technologies such as DataHub, OpenMetadata, or Apache Atlas to improve platform transparency and usability.* Ensure platform security and regulatory compliance through fine-grained access controls, encryption, secrets management, auditing, and secure-by-design engineering principles.Bonus for knowledge of:* Automation/configuration management using toolsets such as Puppet, Chef, Ansible or equivalent* Docker and Kubernetes.* Knowledge and understanding of financial markets.* Setting up CI/CD pipelines using Bamboo.* Confluent certified developer for Apache kafka.* Dollar UniversePersonal Qualities:* Ability to work under pressure with changing priorities, with a view to resolving issues innovatively, and meeting key stakeholders expectations.* A dynamic and self-motivated attitude* Accountable and proactive* Able to provide leadership and motivate team demonstrating strong interpersonal skills* Must display strong analytical skills and attention for detail.* Demonstrates pragmatic judgement, balancing risk and business value to reach decisions which are well informed and actionable.