Lead Data Engineer - GCP Platform & Data Quality

IG Index Limited; City Of London; Hybrid; GBP 110,000 - 140,000; Full time

Benefits offered by this job

Growth opportunities

Mentoring with leaders

Career progression

Volunteer time off

Job summary

IG Group is seeking a Principal Data Engineer to own the technical architecture of its GCP data platform, including BigQuery and Kafka-based ingestion, across London. You will lead delivery of complex, high-risk work in the GCP consolidation programme and define platform contracts and self-service patterns to empower divisions with trusted Core Lake data.

You will mentor engineers, raise data quality standards, and collaborate with Compliance, Finance, and Data Science to deliver scalable,

Qualifications

• Hands-on experience designing and operating large-scale data pipelines on GCP (BigQuery, Cloud Composer/Airflow, GCS) and knowledge of Apache Kafka and real-time streaming architectures.
• Demonstrated experience leading or mentoring a team of data engineers, setting standards, and conducting code/design reviews.
• Strong grasp of data quality practices: data contracts, observability, SLA/SLO definitions, and automated alerting/remediation.

Responsibilities

• Own the technical architecture of IG’s GCP data platform end-to-end — Core Lake (BigQuery), batch and real-time ingestion, and transformation.
• Lead delivery of the most complex, high-risk work within the GCP consolidation programme.
• Define platform contracts, interfaces, and self-service patterns for mature divisions to build on trusted Core Lake data.

Skills

GCP data pipelines

Team leadership

Data quality

SQL & dbt

Stakeholder comms

Tools

BigQuery

Cloud Composer/Airflow

Kafka

Job description

IG Group is seeking a Principal Data Engineer to own the technical architecture of its GCP data platform, including BigQuery and Kafka-based ingestion, across London. You will lead delivery of complex, high-risk work in the GCP consolidation programme and define platform contracts and self-service patterns to empower divisions with trusted Core Lake data.

You will mentor engineers, raise data quality standards, and collaborate with Compliance, Finance, and Data Science to deliver scalable,