Senior Python Data Engineer - AWS Data Pipelines Remote

Halian | Managed Services, Recruitment Agency & Contract Staffing; Manchester; Hybrid; GBP 83,000 - 138,000; Full time

Job summary

Halian | Managed Services, Recruitment Agency & Contract Staffing is seeking a Senior Python Data Engineer to join a major public sector programme. The role supports UK Remote / Hybrid working and focuses on cloud-native data platforms built on AWS with Spark and Airflow.

You will design, build and optimise ETL/ELT pipelines, implement IaC with Terraform, and collaborate with data architects, analysts and engineers to enable analytics and operational decision-making.

Qualifications

• Experience delivering in public sector or highly regulated environments.
• Experience on large-scale cloud data transformation programmes.
• Strong stakeholder engagement and problem-solving skills.
• Ability to work independently in agile delivery teams.

Responsibilities

• Design, develop and maintain scalable data pipelines and ETL/ELT solutions.
• Build and optimise cloud-native data platforms within AWS.
• Develop Spark and PySpark workloads for large-scale data processing.
• Create and maintain Apache Airflow workflows and orchestration pipelines.
• Implement Infrastructure as Code using Terraform.
• Support batch and real-time data ingestion frameworks.
• Work closely with data architects, analysts, data scientists and engineering teams.
• Drive data quality, automation, monitoring and performance optimisation.
• Contribute to CI/CD processes and engineering best practices.

Skills

Python

AWS

Spark

Airflow

Terraform

SQL

ETL / ELT Development

Data Integration

Data Warehousing

Kafka

Docker

Jupyter Notebooks

Snowflake

Tools

Docker

Databricks

Snowflake

Kafka

AWS Glue

EMR

Lambda

Jupyter Notebooks

Job description

Halian | Managed Services, Recruitment Agency & Contract Staffing is seeking a Senior Python Data Engineer to join a major public sector programme. The role supports UK Remote / Hybrid working and focuses on cloud-native data platforms built on AWS with Spark and Airflow.

You will design, build and optimise ETL/ELT pipelines, implement IaC with Terraform, and collaborate with data architects, analysts and engineers to enable analytics and operational decision-making.