Data Engineering Specialist

BT Group; Belfast City District; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

10% on target annual bonus

Private GP 24/7 for you and family

Pension scheme - 5% from you and 10%+

Job summary

BT Group is seeking a Lead Data Engineer in the UK, with Belfast or London basis. The role sits in the Data & AI Enablement Team, delivering high‑quality data products across multi‑cloud environments using AWS, GCP and Ab Initio tooling.

Hybrid work requires 3 days in the office per week, with opportunities to mentor others and drive engineering excellence. You will design and run end‑to‑end pipelines, optimize ETL/ELT, and collaborate with stakeholders to deliver impactful data solutions across

Qualifications

• Must design/build data pipelines and end-to-end data solutions.
• Experience with multi-cloud and big data technologies.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines.
• Build scalable batch and streaming workloads with Spark.
• Develop data ingestion with NiFi for real-time use cases.
• Optimize ETL/ELT for scalability and efficiency.
• Tune Spark jobs, NiFi processors, and Ab Initio graphs for performance.
• Demonstrate and transfer knowledge to stakeholders.
• Mentor junior engineers and share expertise.
• Improve CI/CD pipelines and automated testing.
• Line manage a small team of junior data engineers.
• Continual learning and skill development.

Skills

Big data architecture and design

Data engineering leadership

Agile and stakeholder collaboration

Mentoring junior engineers

Tools

Ab Initio (GDE, PDL, Conduct>IT, Express>IT, EME)

Apache Spark (PySpark/Scala)

Apache NiFi

Hadoop

AWS

GCP

Scala

Java

Python

Kafka

Kinesis

BigQuery

Dataflow

BigTable

Terraform

SQL

CI/CD

Git/SVN

Confluence/JIRA

Job description

All Locations: London, Belfast

Working Style: 3 days a week in office, 2 days from anywhere

About the role

Data is a vital part of our business and with your help, we can improve the services that we provide to our customers through engineering excellence of our data products. You'll be working on exciting data products which support TV, Broadband, Digital Voice and market leading technologies like Unbreakable Wi-Fi. We deliver these critical services using the latest cloud technologies on AWS, GCP and Cloudera.

When you join us as a lead data engineer, you'll have the opportunity to be part of our Data & AI Enablement Team where you will help to design, build, and support high quality engineering products from our extensive data sources and multi-cloud environment.

As a team we believe in the agile principles of openness, transparency and continuous improvement, which underpin strong relationships with our business stakeholders.

There will be lots of opportunities to explore new technologies, develop new skills, innovate and grow as an engineer.

This role can be based in Belfast or London and our hybrid working model means you are expected to be in the office 3 days per week.

What you’ll be doing

• Design, develop, and maintain end-to-end data pipelines using Ab Initio (GDE, PDL, Conduct>IT, Express>IT).
• Build scalable batch and streaming workloads using Apache Spark (PySpark/Scala).
• Develop and orchestrate data ingestion flows using Apache NiFi for real-time and near-real-time use cases.
• Optimize ETL/ELT jobs for scalability, throughput, and resource efficiency.
• Perform bottleneck analysis and tuning for Spark jobs, NiFi processors, and Ab Initio graphs.
• Demos and knowledge transfers to stakeholders around product developments.
• Spread your experience and knowledge to junior engineers.
• Improve automated testing and deployments using CI/CD pipelines.
• Expectation to line manage small team of junior data engineers.
• Continual learning through internal and external training.

Essential Skills / Experience

• Proficient in building Big Data solutions using Hadoop or GCP or AWS.
• Skilled in technologies such as Scala, Java, Python, Spark (PySpark/Scala), Kafka, Kinesis, BigQuery, Dataflow, BigTable, Terraform, and SQL.
• Experience with Terraform, Kafka, Kinesis, SQL are nice to have
• Strong expertise in Ab Initio (GDE, PDL, Conduct>IT, Express>IT, EME) with a focus on parallel processing and component optimisation.
• Hands-on experience with Apache Spark, including Spark SQL, Spark Streaming, and performance tuning.
• Practical experience with Apache NiFi for flow design, processor tuning, error handling, and security configurations.

Desirable Skills / Experience

• Good understanding of Infrastructure as Code toolsets.
• Experienced in implementing Continuous Integration and Deployment strategies.
• Proficient in using Atlassian tools such as Confluence and JIRA.
• Experienced in version control using Git or SVN.
• Able to clearly present technical issues, progress, and outcomes to team members and Product Owners.
• Demonstrates enthusiasm for learning and applying new technologies.
• Experience or interest in managing and developing team members.
• Capable of building new solutions as well as supporting existing data products and platforms.

Our Package

Tailored benefits make a real difference. That's why we offer a comprehensive range to support your growth, wellbeing, and everyday life.

You can design the package to suit you and your lifestyle. Your core benefits include:

• 10% on target annual bonus
• Access to an online private GP 24/7 for you and your immediate family
• Market-leading paid carers leave with up to 2 weeks off
• Equalized maternity, paternity, and adoption leave - 18 weeks' full pay and 8 weeks' half pay
• Discounted EE and BT products, including mobile and broadband
• Market leading Pension scheme - 5% from you and 10% from us
• Holiday purchase scheme

You can select additional benefits, including healthcare, dental, gym memberships and more when you're ready.

BT Group is the UK's leading communications group and the holding company behind some of the country's most recognised brands - including BT, EE, Openreach and Plusnet. Our purpose is as simple as it is ambitious: we connect for good. Our customers include consumers, small, medium and large businesses, public sector organisations and other communications providers.

BT Group's role is about setting direction, unlocking value and creating the conditions for our brands and businesses to thrive.

Having come through the most capital-intensive phase of our fibre investment, our focus now is on what comes next - simplifying how we operate, using technology and AI to work smarter, and organising ourselves to serve customers better and grow sustainably. Group teams shape strategy, policy, brand, capital allocation and transformation, helping the whole organisation perform at its best.

We have a singular culture that unites all our people: we are customer-first challengers, who are committed, clear and connected. These behaviours unite us as one team to deliver for our colleagues, our customers, our stakeholders and the country. Joining BT Group means working at the heart of a business that matters to the UK, with the opportunity to shape decisions, influence outcomes and help set the future course of one of the country's most important companies.