Data Engineer

Blackstone& London; England; Hybrid; GBP 69,000 - 96,000; Full time

Job summary

Blackstone& is seeking a hands-on Data Engineer to join a UK government data programme. You will ingest data into an AWS data lake, build ELT pipelines with dbt and Athena, and ensure data quality for analysts and data scientists.

You will work with analytics engineers, data scientists and source-system teams, using Python, Airflow, Terraform and CI/CD to deliver reliable data products in a hybrid on-site/off-site setup.

Qualifications

• Strong Python and SQL skills.
• Hands-on AWS: S3, Athena, Glue, IAM & DMS
• dbt and Airflow, or close equivalents you can map across quickly
• Software engineering habits: git, code review, automated testing and CI/CD
• Able to explain trade offs to non technical colleagues
• Eligible for SC clearance

Responsibilities

• Ingesting data from relational databases, file transfers and APIs into an S3 based data lake
• Building ELT pipelines with dbt and Amazon Athena, using Apache Iceberg tables
• Writing and scheduling workflows in Apache Airflow
• Using Python for pre processing, validation and automation, with tests in pytest
• Managing infrastructure as code with Terraform, and shipping through GitHub Actions CI/CD
• Building data quality checks so issues are caught before users see them
• Helping analysts and data scientists find, understand and use the data you deliver
• Keeping documentation current, so others can pick up your work

Skills

Python

SQL

AWS

dbt

Airflow

Git

CI/CD

Trade-offs

SC clearance

Tools

Terraform

Docker

Kubernetes

Lambda

API Gateway

Spark

Glue

Job description

Data Engineer - AWS Data Platform, Public Sector - SC eligible

About the role

Blackstone& is looking for a hands on Data Engineer to join a UK government data programme. The client runs an AWS data lake that analysts and data scientists rely on every day, and is moving more operational systems onto it. You'll help get data in reliably, model it well and keep it trustworthy.

You'll spend most of your time building, testing and running pipelines, working closely with analytics engineers, data scientists and the teams that own the source systems.

What you'll be doing

• Ingesting data from relational databases, file transfers and APIs into an S3 based data lake
• Building ELT pipelines with dbt and Amazon Athena, using Apache Iceberg tables
• Writing and scheduling workflows in Apache Airflow
• Using Python for pre processing, validation and automation, with tests in pytest
• Managing infrastructure as code with Terraform, and shipping through GitHub Actions CI/CD
• Building data quality checks so issues are caught before users see them
• Helping analysts and data scientists find, understand and use the data you deliver
• Keeping documentation current, so others can pick up your work

What we're looking for

• Strong Python and SQL
• Hands on AWS: S3, Athena, Glue, IAM & DMS
• dbt and Airflow, or close equivalents you can map across quickly
• Software engineering habits: git, code review, automated testing and CI/CD
• Able to explain trade offs to non technical colleagues
• Eligible for SC clearance

Nice to have:

• Terraform, Docker or Kubernetes
• Lambda or API Gateway for ingestion
• Spark or Glue PySpark for heavier workloads
• Experience in government or another highly regulated sector

Why this role

The stack is modern and open source, and the programme has backing to grow. The data you deliver will be used by analysts across the organisation to inform major public decisions.

ENGAGEMENT DETAILS

This engagement is offered outside IR35, reflecting the genuine independence and expertise the role requires. Key terms:

• Rate: Competitive
• Outside IR35 - engaged as an independent contractor via your own limited company or umbrella
• Initial contract length: 6 months, with extensions expected subject to performance and project progression
• Hybrid working - a mix of remote and on-site with the client as the engagement requires
• Start date: to be agreed. We are looking to move quickly for the right person