Principal Data Engineer

HKEX Group; Greater London; Hybrid; GBP 120,000 - 200,000; Full time

Job summary

HKEX Group in London seeks a Principal Data Engineer to shape a strategic, scalable data platform for high-volume trading data. The role focuses on building a resilient data foundation, enabling rapid, self-serve data access for engineering teams and analytics workloads.

You will influence technology strategy, drive platform reliability and performance, and work across Trading Technology, Data Engineering and Platform Engineering to deliver a data-driven trading infrastructure.

Qualifications

• Ten+ years of hands-on software development experience in IT organisations.
• Expertise in scalable data platforms, data warehouses (SQL Server, Postgres) and data lakes.
• Proven ability to solve performance, scalability and latency challenges in large distributed systems.
• Self-service data platform capabilities enabling onboarding, processing, discovery and consumption of data.
• Influences tech strategy and roadmaps across Trading Tech, Data Eng and Platform Eng.
• Builds high-volume platforms handling billions of events daily and terabytes of data.
• Experience with Data Lakehouse using S3 storage, Iceberg, Spark and Trino.
• Real-time streaming with Kafka and Flink for low-latency processing.
• Strong in CI/CD, GitOps, Kubernetes/OpenShift, Terraform and IAM/security best practices.
• Knowledge of data lineage, metadata and governance tools (DataHub/OpenMetadata/Atlas).
• Understanding of financial markets; Bamboo CI/CD knowledge is a bonus; Kafka certification is a plus.

Responsibilities

• Shape a scalable data platform to reliably process, store and serve high-volume trading data.
• Provide self-service capabilities for engineering teams to onboard and use data effectively.
• Lead platform engineering efforts to improve reliability, performance and security.
• Design APIs and data products with metadata services for developer enablement.
• Drive platform-wide CI/CD and infrastructure as code practices and governance.
• Collaborate across Trading Tech, Data Eng and Platform Eng to set roadmaps.

Skills

Java/Scala/Kotlin

Distributed data platforms

Data lakes

PostgreSQL/SQL Server

Kafka/Stream processing

Spark/Trino

Iceberg/Data Lakehouse

Kubernetes/OpenShift

Terraform/CI-CD

Data governance/Metadata

Tools

DataHub

OpenMetadata

Apache Atlas

Apache Kafka

Apache Flink

Spark

Trino

Iceberg

Job description

Principal Data Engineer

Shift Pattern: Standard 40 Hour Week (United Kingdom) Scheduled Weekly Hours: 40 Corporate Grade: C - Vice President Reporting Line: (UK Division) Information Technology Location: UK-London Worker Type: Permanent

The purpose of this role is to help shape the LME’s strategic data platform which can reliably process, store, and serve high-volume, high-velocity trading data at scale, while providing engineering teams with reusable, self-service capabilities that accelerate the delivery of trading and analytical solutions.

Success in this role means delivering a highly scalable and resilient data platform that becomes the engineering foundation for all data-intensive trading workloads, enabling faster innovation, improved reliability, reduced operational complexity, and better utilisation of market and trading data across the organisation.

Strong Required Demonstrable Experience of:

• A candidate for this position must have got working experience in IT organisation as a technologist who has evolved from a strong background in hands on software development: 10+ years of experience with software development, preferably working with JVM based languages (Java/Scala/Kotlin).
• Provide expertise on the development of highly scalable data platforms, data warehouses (SQL server, Postgres), and modern data lakes.
• Solve complex scalability, reliability, latency, and performance challenges associated with large-scale distributed systems.
• Provide self-service platform capabilities that allow engineering teams to onboard, process, discover, and consume data efficiently.
• Influence technology strategy and roadmaps across Trading Technology, Data Engineering, and Platform Engineering functions.
• Build high-volume distributed data platforms capable of processing billions of events per day and managing terabytes scale datasets.
• Develop modern Data Lakehouse platforms leveraging S3-compatible object storage, Apache Iceberg, Spark, and Trino.
• Build real-time streaming integration using Apache Kafka and Apache Flink for low-latency event processing, enrichment, and data distribution.
• Drive performance engineering and scalability across data ingestion, storage, streaming, query, and analytics platforms.
• Design and build self-service data platform capabilities including APIs, data products, metadata services, and developer enablement frameworks.
• Implement cloud-native platform engineering practices using Kubernetes/OpenShift, Terraform, CI/CD, GitOps, and Infrastructure as Code.
• Lead metadata, lineage, and data discovery capabilities using technologies such as DataHub, OpenMetadata, or Apache Atlas to improve platform transparency and usability.
• Ensure platform security and regulatory compliance through fine-grained access controls, encryption, secrets management, auditing, and secure-by-design engineering principles.
• Bonus for knowledge of: Automation/configuration management using toolsets such as Puppet, Chef, Ansible or equivalent Docker and Kubernetes.
• Knowledge and understanding of financial markets.
• Setting up CI/CD pipelines using Bamboo.
• Confluent certified developer for Apache kafka.

Dollar Universe Personal Qualities:

• Ability to work under pressure with changing priorities, with a view to resolving issues innovatively, and meeting key stakeholders expectations.
• A dynamic and self-motivated attitude.
• Accountable and proactive.
• Able to provide leadership and motivate team demonstrating strong interpersonal skills.
• Must display strong analytical skills and attention for detail.
• Demonstrates pragmatic judgement, balancing risk and business value to reach decisions which are well informed and actionable.

Hong Kong Exchanges and Clearing Limited (HKEX) is a publicly-traded company (HKEX Stock Code:388) and one of the world’s leading global exchange groups, offering a range of equity, derivative, commodity, fixed income and other financial markets, products and services, including the London Metals Exchange. As a superconnector and gateway between East and West, HKEX facilitates the two-way flow of capital, ideas and dialogue between China and the rest of world, through its pioneering Connect schemes, increasingly diversified product ecosystem and its deep, liquid and international markets. HKEX is a purpose-led organisation which, across its business and through the work of HKEX Foundation, seeks to connect, promote and progress its markets and the communities it supports for the prosperity of all. Discover the latest career opportunities and programmes at HKEX.