Lead Data Engineer – Multi-Cloud Pipelines & AI Enablement

BT Group; Belfast City District; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

10% on target annual bonus

Private GP 24/7 for you and family

Pension scheme - 5% from you and 10%+

Job summary

BT Group is seeking a Lead Data Engineer in the UK, with Belfast or London basis. The role sits in the Data & AI Enablement Team, delivering high‑quality data products across multi‑cloud environments using AWS, GCP and Ab Initio tooling.

Hybrid work requires 3 days in the office per week, with opportunities to mentor others and drive engineering excellence. You will design and run end‑to‑end pipelines, optimize ETL/ELT, and collaborate with stakeholders to deliver impactful data solutions across

Qualifications

• Must design/build data pipelines and end-to-end data solutions.
• Experience with multi-cloud and big data technologies.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines.
• Build scalable batch and streaming workloads with Spark.
• Develop data ingestion with NiFi for real-time use cases.
• Optimize ETL/ELT for scalability and efficiency.
• Tune Spark jobs, NiFi processors, and Ab Initio graphs for performance.
• Demonstrate and transfer knowledge to stakeholders.
• Mentor junior engineers and share expertise.
• Improve CI/CD pipelines and automated testing.
• Line manage a small team of junior data engineers.
• Continual learning and skill development.

Skills

Big data architecture and design

Data engineering leadership

Agile and stakeholder collaboration

Mentoring junior engineers

Tools

Ab Initio (GDE, PDL, Conduct>IT, Express>IT, EME)

Apache Spark (PySpark/Scala)

Apache NiFi

Hadoop

AWS

GCP

Scala

Java

Python

Kafka

Kinesis

BigQuery

Dataflow

BigTable

Terraform

SQL

CI/CD

Git/SVN

Confluence/JIRA

Job description

BT Group is seeking a Lead Data Engineer in the UK, with Belfast or London basis. The role sits in the Data & AI Enablement Team, delivering high‑quality data products across multi‑cloud environments using AWS, GCP and Ab Initio tooling.

Hybrid work requires 3 days in the office per week, with opportunities to mentor others and drive engineering excellence. You will design and run end‑to‑end pipelines, optimize ETL/ELT, and collaborate with stakeholders to deliver impactful data solutions across