Lead Data Engineer

Blackstone& London; United Kingdom; Hybrid; GBP 111,000 - 166,000; Part time

Job summary

Blackstone is seeking a Lead Data Engineer to guide the government AWS data programme. You will lead a small team, shape data intake and stewardship, and stay hands-on with critical pipelines.

The role involves partnering with stakeholders, designing integration paths, and establishing standards for testing, CI/CD and data quality. Hybrid, contract-based engagement.

Qualifications

• Leads data engineering teams and makes technical decisions that endure.
• Expert in Python and SQL.
• Deep AWS experience with S3, Athena, Glue, IAM and related services.
• Production experience with dbt and Airflow or equivalents.
• Strong software engineering practices: pytest, git, GitHub Actions or similar CI/CD, IaC.

Responsibilities

• Lead and line manage a team of data engineers; grow their skills through pairing and coaching.
• Design how new source systems integrate into the platform from batch extracts to API feeds.
• Set engineering standards for testing, CI/CD, documentation, and data quality.
• Make architecture decisions across S3, Athena, Glue, Iceberg, dbt and Airflow; document rationale.
• Own infrastructure as code in Terraform and keep deployments repeatable.
• Agree scope and delivery dates with product owners and senior data users.
• Replace manual pipelines with automated, monitorable ones.
• Represent data engineering in architecture and governance forums.

Skills

Python

SQL

AWS

Leadership

CI/CD

Terraform

Airflow

Communication

Tools

dbt

GitHub Actions

Terraform

Docker

Kubernetes

Job description

Lead Data Engineer - Government AWS Data Programme - SC eligible

About the role

Blackstone& is placing a Lead Data Engineer with a UK government department to set the technical direction for its data engineering work. The organisation runs an AWS data lake used by analysts and data scientists across the organisation. It needs someone to raise the standard of how data gets onto it and how it is looked after once it's there.

You'll lead a small team of data engineers and stay hands-on. You'll share the work, review code, unblock people and still write a fair amount yourself.

What you'll be doing

• Leading and line managing a team of data engineers, and growing their skills through pairing, review and coaching
• Designing how new source systems are integrated into the platform, from batch database extracts to API feeds
• Setting engineering standards for testing, CI/CD, documentation and data quality, and making sure they're followed
• Making architecture decisions across S3, Athena, Glue, Iceberg, dbt and Airflow, and writing them down so others can follow the reasoning
• Owning infrastructure as code in Terraform and keeping deployments repeatable
• Agreeing scope, priorities and delivery dates with product owners, system owners and senior data users
• Replacing manual or fragile pipelines with automated ones that can be monitored
• Representing data engineering in wider architecture and governance forums

What we're looking for

• A track record of leading data engineering teams and making technical decisions that held up
• Expert Python and SQL
• Deep AWS experience: S3, Athena, Glue, IAM and related data services
• Production experience with dbt and Airflow, or close equivalents
• Strong software engineering practice: pytest, git, GitHub Actions or similar CI/CD, and infrastructure as code (Terraform)
• Experience integrating complex operational systems into an analytics platform
• Confidence managing stakeholders up to senior level, including pushing back when needed
• Eligible for SC clearance

Nice to have:

• Apache Iceberg, the medallion (bronze/silver/gold) pattern, or Write-Audit-Publish
• Docker, Kubernetes or Spark
• Line management in a consultancy or client-facing setting
• Experience in government or another highly regulated sector, including data protection and audit requirements

Why this role

You'll have real influence over how a large public-sector data platform grows. Your team's work will reach decision-makers across the organisation.

ENGAGEMENT DETAILS

This engagement is offered outside IR35, reflecting the genuine independence and expertise the role requires. Key terms:

• Rate: Competitive
• Outside IR35 - engaged as an independent contractor via your own limited company or umbrella
• Initial contract length: 6 months, with extensions expected subject to performance and project progression
• Hybrid working - a mix of remote and on-site with the client as the engagement requires
• Start date: to be agreed. We are looking to move quickly for the right person