Data Engineer: AWS Data Lake for Public Sector

Blackstone& London

England

Hybrid

GBP 69,000 - 96,000

Full time

11 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Blackstone& is seeking a hands-on Data Engineer to join a UK government data programme. You will ingest data into an AWS data lake, build ELT pipelines with dbt and Athena, and ensure data quality for analysts and data scientists.

You will work with analytics engineers, data scientists and source-system teams, using Python, Airflow, Terraform and CI/CD to deliver reliable data products in a hybrid on-site/off-site setup.

Qualifications

  • Strong Python and SQL skills.
  • Hands-on AWS: S3, Athena, Glue, IAM & DMS
  • dbt and Airflow, or close equivalents you can map across quickly
  • Software engineering habits: git, code review, automated testing and CI/CD
  • Able to explain trade offs to non technical colleagues
  • Eligible for SC clearance

Responsibilities

  • Ingesting data from relational databases, file transfers and APIs into an S3 based data lake
  • Building ELT pipelines with dbt and Amazon Athena, using Apache Iceberg tables
  • Writing and scheduling workflows in Apache Airflow
  • Using Python for pre processing, validation and automation, with tests in pytest
  • Managing infrastructure as code with Terraform, and shipping through GitHub Actions CI/CD
  • Building data quality checks so issues are caught before users see them
  • Helping analysts and data scientists find, understand and use the data you deliver
  • Keeping documentation current, so others can pick up your work

Skills

Python
SQL
AWS
dbt
Airflow
Git
CI/CD
Trade-offs
SC clearance

Tools

Terraform
Docker
Kubernetes
Lambda
API Gateway
Spark
Glue

Job description

Blackstone& is seeking a hands-on Data Engineer to join a UK government data programme. You will ingest data into an AWS data lake, build ELT pipelines with dbt and Athena, and ensure data quality for analysts and data scientists.

You will work with analytics engineers, data scientists and source-system teams, using Python, Airflow, Terraform and CI/CD to deliver reliable data products in a hybrid on-site/off-site setup.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

Blackstone& London • England

Hybrid
GBP 69,000 - 96,000
Senior Data Engineer - Remote (Public Sector, AWS & Spark)
Senior Data Engineer - Remote (Public Sector, AWS & Spark)

Modular Data • United Kingdom

Remote
GBP 70,000 - 110,000
Fully remote (UK-based)
Data Engineer
Data Engineer

Modular Data • United Kingdom

Remote
GBP 70,000 - 110,000
Fully remote (UK-based)
AWS Data Engineer
AWS Data Engineer

RED Global • Basildon

On-site
GBP 65,000 - 90,000
AWS Data Engineer
AWS Data Engineer

Red Commerce Schweiz GmbH • Essex

On-site
GBP 92,000 - 120,000
Databricks Data Engineer
Databricks Data Engineer

GIS Recruiter Ltd • Greater London

On-site
GBP 90,000 - 120,000
AWS Data Engineer (PySpark/AI) - Hybrid London
AWS Data Engineer (PySpark/AI) - Hybrid London

Datatech Analytics • City Of London

Hybrid
GBP 45,000 - 75,000
Databricks Data Engineer
Databricks Data Engineer

GIS Recruiter Ltd • City Of London

On-site
GBP 90,000 - 130,000
Senior Data Engineer - Remote (Public Sector)
Senior Data Engineer - Remote (Public Sector)

Modular Data • United Kingdom

Remote
GBP 101,000 - 166,000
Outside IR35 contract
Fully remote (UK-based)
Meaningful public sector projects
+1
Data Engineer (contract)
Data Engineer (contract)

Modular Data • United Kingdom

Remote
GBP 101,000 - 166,000
Outside IR35 contract
Fully remote (UK-based)
Meaningful public sector projects
+1