Software Engineer

Halian | Managed Services, Recruitment Agency & Contract Staffing; Manchester; Hybrid; GBP 83,000 - 138,000; Full time

Job summary

Halian | Managed Services, Recruitment Agency & Contract Staffing is seeking a Senior Python Data Engineer to join a major public sector programme. The role supports UK Remote / Hybrid working and focuses on cloud-native data platforms built on AWS with Spark and Airflow.

You will design, build and optimise ETL/ELT pipelines, implement IaC with Terraform, and collaborate with data architects, analysts and engineers to enable analytics and operational decision-making.

Qualifications

• Experience delivering in public sector or highly regulated environments.
• Experience on large-scale cloud data transformation programmes.
• Strong stakeholder engagement and problem-solving skills.
• Ability to work independently in agile delivery teams.

Responsibilities

• Design, develop and maintain scalable data pipelines and ETL/ELT solutions.
• Build and optimise cloud-native data platforms within AWS.
• Develop Spark and PySpark workloads for large-scale data processing.
• Create and maintain Apache Airflow workflows and orchestration pipelines.
• Implement Infrastructure as Code using Terraform.
• Support batch and real-time data ingestion frameworks.
• Work closely with data architects, analysts, data scientists and engineering teams.
• Drive data quality, automation, monitoring and performance optimisation.
• Contribute to CI/CD processes and engineering best practices.

Skills

Python

AWS

Spark

Airflow

Terraform

SQL

ETL / ELT Development

Data Integration

Data Warehousing

Kafka

Docker

Jupyter Notebooks

Snowflake

Tools

Docker

Databricks

Snowflake

Kafka

AWS Glue

EMR

Lambda

Jupyter Notebooks

Job description

Senior Python Data Engineer | AWS | Spark | Airflow | Public Sector

Location: UK Remote / Hybrid

Rate: Competitive Daily Rate

Duration: Long-Term Contract

Sector: Public Sector

We are supporting a major public sector programme and are looking for an experienced Senior Data Engineer to join a high-performing cloud data team delivering business-critical data solutions.

About the Role

This is an opportunity to work on large-scale data platforms, helping design, build and optimise modern cloud-native data services that support analytics, reporting and operational decision-making.

Responsibilities

• Design, develop and maintain scalable data pipelines and ETL/ELT solutions
• Build and optimise cloud-native data platforms within AWS
• Develop Spark and PySpark workloads for large-scale data processing
• Create and maintain Apache Airflow workflows and orchestration pipelines
• Implement Infrastructure as Code using Terraform
• Support batch and real-time data ingestion frameworks
• Work closely with data architects, analysts, data scientists and engineering teams
• Drive data quality, automation, monitoring and performance optimisation
• Contribute to CI/CD processes and engineering best practices

Qualifications

Experience Required

• Previous experience delivering within public sector or highly regulated environments
• Experience working on large-scale cloud data transformation programmes
• Strong stakeholder engagement and problem-solving skills
• Ability to work independently and contribute within agile delivery teams

Required Skills

• Python
• AWS
• Terraform
• SQL
• ETL / ELT Development
• Data Integration
• Data Warehousing

Preferred Skills

• AWS Glue
• EMR
• Lambda
• Databricks
• Kafka
• Docker
• Jupyter Notebooks
• Snowflake
• Data Governance & Data Quality Frameworks

Pay range and compensation package

• Rate: Competitive Daily Rate
• Duration: Long-Term Contract

Equal Opportunity Statement

We welcome applications from all suitably qualified candidates and are committed to creating an inclusive environment for all employees, including those working within public sector or highly regulated environments.