Data Engineer (SEO)

Infected Blood Compensation Authority; Newcastle upon Tyne; On-site; GBP 48,000; Full time

Job summary

A UK public body is seeking a Data Engineer in Newcastle upon Tyne. The role involves leading data engineering efforts, ensuring data quality, and mentoring a team. Candidates should have strong experience in AWS, Databricks, and data pipeline management. The position supports a culture of diversity and flexibility, offering comprehensive benefits including a competitive salary and a Civil Service pension scheme. Be part of a mission that impacts those affected by the infected blood scandal.

Benefits

Civil Service pension scheme

32.5 days annual leave

Flexible working options

Qualifications

• Strong experience in building and maintaining complex data pipelines.
• Experience with structured and unstructured data in cloud environments.
• Demonstrable experience of setting up data engineering processes.

Responsibilities

• Lead the data engineering team to develop robust data solutions.
• Design and deliver scalable data platforms using AWS.
• Mentor junior colleagues and foster a strong data engineering community.

Skills

Data analysis and synthesis

Cloud-native data engineering (AWS)

Strong problem-solving skills

Experience with ETL/ELT pipelines

Proficiency in Python

Education

Graduate degree or equivalent experience in a relevant field

Tools

Databricks

Apache Spark

Job description

About IBCA

The Infected Blood Compensation Authority (IBCA) is a new arm's‑length body set up at an unprecedented pace to administer compensation to people whose lives have been impacted by the infected blood scandal. IBCA will ensure payment is made in recognition of the wrongs experienced by those who have been infected by HIV, Hepatitis B or C, as well as those who love and care for them. They have been frustrated and distressed by the delays in achieving proper recognition, and we must help put this right.

We are committed to putting the infected and affected blood community at the centre of every decision we make and every step we take to build our organisation to deliver compensation payments. IBCA employees will be public servants. If successful in this role you will be appointed directly into IBCA, on IBCA terms and conditions as a public servant. Successful applicants will join the Civil Service Pension Scheme.

Please note that the mission of IBCA means that it is likely to be operational for a period of approximately 5 to 7 years. When IBCA's work begins to wind down, IBCA employees will receive support and practical guidance to find a new role, whether in the Civil Service, another Arms Length Body (ALB), or an external employer.

Role Overview

This role will lead our data engineering capability within the engineering team of the Data Operations arm of the IBCA Data Directorate. The Data Operations team is responsible for developing and running safe and secure data solutions that provide a single source of truth for those going through their compensation journey. We are building a new data platform using Amazon Web Services (AWS) and data management and intelligence products using Databricks, Quantexa and Tableau. We are taking a product‑centric approach treating data as a product and are building squads around our products, with a focus on paying compensation to those impacted by the infected blood scandal seamlessly.

You will focus on building robust data pipelines and data management processes (including master data management), ensuring data quality, and driving a culture of data‑driven decision‑making. Working at IBCA gives you a huge opportunity to make an impact on those who deserve compensation, and this role suits a candidate who can lead a team of data engineers to deliver solutions from the ground up to take from ideation to reality so that data is an enabler to everything IBCA does.

As a Data Engineer, you will join a multidisciplinary team to design, build, and deliver high‑impact, scalable AWS‑based data solutions using Databricks and Quantexa technologies, while also driving data quality, mentoring junior colleagues, and fostering a strong data engineering community.

Responsibilities

• Provide technical guidance for the development of robust, automated data pipelines and master data management processes for our data platform and products, encompassing DevSecOps best practices.
• Ensure tools and techniques are scalable, secure and efficient.
• Be responsible for ensuring the right data engineering practices are embedded consistently and to industry standard best practices within the data platform delivery teams.
• Further develop your own data engineering and leadership skills.
• Work with business stakeholders and across digital service teams, understanding their needs and translating them into data development.
• Design, build, and deliver high‑performance, scalable, and secure data solutions in a complex data environment within AWS, Databricks and Quantexa. This includes building robust ETL/ELT pipelines, and ensuring seamless integration between platforms.
• Collaborate extensively with multidisciplinary teams across product delivery, architecture, engineering security and analytics to understand and meet data needs.
• Drive high build and data quality, ensuring stability, robustness, and resilience of products, and embed Agile, CI/CD and DevOps practices.
• Mentor junior team members, foster professional development, build a data engineering community, review solutions, troubleshoot complex issues and proactively drive innovation and challenge existing methods.
• Communicate complex data solutions and champion innovative approaches to diverse stakeholders, while also anticipating and resolving intricate data engineering challenges and building consensus to align data designs with organisational goals.

Behaviours

• Changing and Improving
• Communicating and Influencing

Technical Skills

• Data analysis and synthesis
• Data development process
• Strong experience of cloud‑native data engineering in AWS and building and maintaining complex data pipelines (both ETL and ELT) in a rapid delivery setting that covers data quality & transformation processes, data matching and master data management.
• Experience working with structured and unstructured data, and data lakes to service operational and analytical business needs.
• Proficiency in writing clear parameterised code in two or more of the following – Python, Databricks, Apache Spark (PySpark, Spark SQL), NoSQL, Scala.
• Experience delivering through Agile/DevOps working practices in multi‑disciplinary teams – CI/CD, Scrum, Automation.
• Strong problem‑solving skills, including assessing and mitigating risks while identifying opportunities for innovation.
• Experience in the full end‑to‑end data lifecycle for design, build and test, with knowledge of the interactions and dependencies with data architecture, data modelling and testing engineering.
• Demonstrable experience of setting up data engineering processes from scratch or implementing large changes to existing processes within an organisation that operate in a cloud DevSecOps environment.

Benefits

• Salary of £47,258.
• IBCA contributes £13,690 towards you being a member of the Civil Service Defined Benefit Pension scheme.
• Premium allowance paid monthly after probation.
• Learning and development tailored to your role.
• Environment with flexible working options.
• Culture encouraging inclusion and diversity.
• 32.5 days of paid annual leave plus 8 bank holidays.
• Family friendly policies to support you and your everyday responsibilities.
• Enhanced maternity and paternity leave, up to 12 months shared parental leave.

Additional Information

This vacancy is using Success Profiles and will assess your Behaviours, Experience and Technical skills. Recent changes in skilled worker visa eligibility mean that a Skilled Worker must have a job offer in an eligible skilled occupation from a Home Office‑approved sponsor. From 22 July 2025, the role must normally be skilled to level 6 (graduate level) on the Regulated Qualifications Framework for England and Northern Ireland, or the equivalent level in Wales or Scotland, or be included on either the Immigration Salary List or the Temporary Shortage List. If the role is not eligible for a skilled worker visa, we will be unable to provide sponsorship.