Data Engineer

Behavioural Insights Team; Manchester; Hybrid; GBP 42,000 - 55,000; Full time

Job summary

Join the Behavioural Insights Team as a Data Engineer to develop innovative solutions using your Python skills. Contribute to impactful projects by maintaining our top-tier data platform and ensure robust data pipelines while working in a collaborative and flexible environment.

Benefits

Generous pension scheme with 7% employer contribution

Flexible benefits programme with £1,100 flexible benefits pot

25 days of annual leave plus public holidays

Enhanced parental leave payments

Culture promoting flexible working

Team lunches with guest speakers

Qualifications

• Practical application of Python in professional settings.
• Experience with data pipelines and infrastructure.
• Strong knowledge of Python Data Science stack.

Responsibilities

• Lead development and maintenance of Python data pipelines.
• Contribute to building and maintaining Predictiv platform.
• Collaborate to deliver data-driven solutions.

Skills

Python

Data Engineering

ETL/ELT

Data Governance

Cloud Platforms

Job description

Available on a full time or part-time basis (39 hours or 32 hours per week/ 4-5 days) Flexible working options include flexible start and finish times and some home working. Starting salary: from £42,000 per annum FTE (depending on experience), plus benefits and bonus. Based in Central London (Blackfriars) and Manchester. We use a hybrid model with 2 days per week minimum in the office.

Are you looking for a new challenge and want to join a successful and rapidly growing organisation? Do you have a passion for enabling people to perform at their best? If so, we want to hear from you.

We are looking for a Data Engineer (Python) to join our Data Science and Technology team at BIT. As a Data Engineer you will be responsible for the technical delivery of our digital products and client solutions, working closely with our Group Product Lead and Full Stack Developer to ensure we are building and maintaining robust, scalable technology. You will play a crucial role in maintaining and enhancing Predictiv – our online research platform running at production scale – while delivering innovative data engineering solutions for client-facing projects.

This is an exciting opportunity to use your technical skills while contributing to meaningful, high-impact projects. You will gain hands-on experience with modern data engineering tools, cloud infrastructure, and collaborative data science workflows, all while working in a fast paced environment focussed on driving social impact.

About the Behavioural Insights Team (BIT)

Behavioural Insights Team (BIT) is a social purpose company. It exists to improve people’s lives and communities. It works in partnership with national, regional and local governments, businesses, foundations and charities, to tackle major policy problems, often through testing and implementing simple yet powerful changes.

It has 250 staff based in 8 countries across Europe, the Americas and Asia Pacific, and has run more than 1000 projects to date, including over 700 randomised controlled trials in dozens of countries. In addition to its own interventions, it openly shares expertise and helps others build capacity and skills to apply behavioural science for social good.

What does a Data Engineer at BIT do?

• Contribute to forward development and lead maintenance of Predictiv – our industry-best research platform.
• Lead on the development and maintenance of Python data pipelines to collect, clean, and process data at scale, ensuring robustness, reliability, and efficiency.
• Enable efficient model training and deployment.
• Collaborate across teams to deliver outcome-focused, data-driven solutions and products that generate revenue and deliver social impact.
• Help to build applied research infrastructure and tooling that allows our teams to perform scalable and robust research activities.
• Steward our data governance processes, enabling our mission teams to make impactful use of our data assets.

What makes a great Data Engineer at BIT?

• Demonstrated practical application of Python in professional settings – using the language consistently in data-centric projects.
• Experience in developing and maintaining pipelines and infrastructure for products or applications, at scale.
• Strong knowledge of the Python Data Science stack (e.g., pandas/polars, scikit-learn).
• Ability to independently develop and maintain robust Python-based ETL/ELT data pipelines.
• Ability to independently develop LLM-based tools/products (e.g., RAG workflows).
• Familiarity with version control tools such as Git/GitHub.

In addition, we are particularly interested to hear from candidates who have:

• Experience setting up cloud platforms (e.g., AWS, GCP, or Azure).
• An understanding of DevOps concepts such as Infrastructure as Code, CI/CD, or MLOps.
• Knowledge of Python-based data pipeline frameworks (e.g. Airflow, Metaflow) and Python-based product development packages (e.g. FastAPI, Streamlit).
• An interest in developing or improving full stack product development skills.

Who are we and what can we offer you?

The Behavioural Insights Team is the world leader in applying behavioural science to public policy challenges in order to strengthen communities and improve lives.

We design and rigorously test new ideas across a range of areas such as health, crime and education, to establish what works and what doesn’t. We have helped Tuberculosis patients take their medicine; reduced police susceptibility to cyberattacks; and helped organisations close their gender pay gap.

BIT really is a great place to work:

• Generous pension scheme with 7% employer contribution
• A flexible benefit pot of £1,100 to spend on our flexible benefits programme which currently includes things like the ability to buy extra annual leave at the start of the year, gym membership, cycle to work scheme etc or just take as cash.
• 25 days of annual leave (in addition to public holidays) for full-time employees, rising to 27 days after 2 years continuous service
• We offer enhanced parental leave payments
• We encourage flexible working and have a culture which enables this; for example flexible start and finish times and an IT set up which allows for easy working from home options
• We have regular team lunches with inspirational guest speakers.
• We’re a social bunch and have a variety of interest groups including Film Club, BIT Kids for our parents & 5-a-side team to name a few.
• We have a great office, which has a fabulous roof terrace overlooking the Thames river, an amazing coffee machine and fridges stocked full of soft drinks!

How to apply

We use a great platform called Applied to reduce bias in the recruitment process. You will need to answer a few work-based questions which will be anonymously marked by members of our team. Our selection is based on performance in these questions, as well as reviewing your CV to understand your experience.

Candidates with the top scoring answers will be invited to a structured interview where you get to meet some of the team. At the interview stage you will be asked questions that reflect the role and also to complete a work-based task, so you have the chance to demonstrate your skills. It’s at this stage we first see your CV. We always give all applicants feedback, no matter what stage your application reaches.

Please note:

You will need to have the right to work in the UK in order to be able to undertake this role. We will ask all applicants short-listed for an interview to provide evidence of their right to work during the recruitment process. If you do not have the right to work for BIT but are the most suitable person for the role we will explore with you whether sponsorship may be an option. However, this may not be possible for all roles.

Some of the personal data you provide to Applied when you apply for a role with The Behavioural Insights Team will be passed on to The Behavioural Insights Team for the purposes of (a) reviewing your application; (b) inviting you for interview; and/or (c) offering you a job and getting you set up as an employee/contractor.For full details on how BIT collects and uses your personal data please visit our website – https://www.bi.team/bit-global-recruitment-privacy-notice/

If your application is not successful, any personal data provided to us by Applied (including any personal data relating to equal opportunities monitoring as set out above) will be retained by The Behavioural Insights Team for no longer than 6 months after the application deadline has passed (other than in exceptional circumstances where we may need to retain your personal data for longer), at which point your personal data will be securely deleted. We only keep your information for that period in order to allow us to comply with relevant legal obligations.