AWS Data Engineer - Airflow

Square One Resources; Ringway; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A leading financial services client is looking for an Airflow/AWS Data Engineer to design, deploy, and maintain cloud infrastructure and data pipelines. This contract role requires expertise in AWS services and Airflow with a focus on cloud best practices. Ideal candidates will have strong skills in automation, data workflows, and CI/CD integration.

Qualifications

• Proven experience as an AWS DevOps Engineer or Data Engineer in complex cloud environments.
• Strong hands-on expertise with AWS services.
• Demonstrated experience with Airflow setup and optimization.

Responsibilities

• Deploy and manage cloud infrastructure across Airflow and AccelData environments.
• Integrate vendor products with core systems including data lakes.
• Implement monitoring, alerting, and logging solutions for Airflow.

Skills

AWS services (EC2, S3, Lambda, RDS, IAM, CloudWatch)

Airflow (Astronomer)

Infrastructure as code (IaC) tools

CI/CD pipelines

Containerization technologies (Docker, Kubernetes)

Python

SQL

Monitoring and observability tools (Grafana, Prometheus)

Job description

Overview

Job Title: Airflow/AWS Data Engineer
Location: Manchester Area (3 days per week in the office)
Rate: Up to £400 per day inside IR35
Start Date: 03/11/2025
Contract Length: Until 31st December 2025
Job Type: Contract

Company Introduction

An exciting opportunity has become available with one of our sector-leading financial services clients. They are seeking a talented AWS DevOps/Data Engineer to join their growing data engineering function. This role will play a key part in designing, deploying, and maintaining modern cloud infrastructure and data pipelines, with a focus on Airflow, AWS, and data platform automation.

Responsibilities

• Deploy and manage cloud infrastructure across Astronomer Airflow and AccelData environments.
• Facilitate integration between vendor products and core systems, including data lakes, storage, and compute services.
• Establish and enforce best practices for cloud security, scalability, and performance.
• Configure and maintain vendor product deployments, ensuring reliability and optimized performance.
• Ensure high availability and fault tolerance for Airflow clusters.
• Implement and manage monitoring, alerting, and logging solutions for Airflow and related components.
• Perform upgrades, patches, and version management for platform components.
• Oversee capacity planning and resource optimization for Airflow workers and AWS resources.
• Manage integrations with source control systems (GitHub, GitLab) and CI/CD pipelines.
• Collaborate with AWS teams and internal stakeholders for pipeline scalability and optimization.
• Design and implement process improvements, including automation, data delivery optimization, and infrastructure re-design.
• Develop ETL pipelines and data workflows using AWS and SQL technologies.
• Partner with cross-functional teams (product, design, and leadership) to resolve technical issues and enhance platform capabilities.
• Build analytical tools and dashboards to leverage data pipelines for actionable business insights.

Key Requirements

• Proven experience as an AWS DevOps Engineer or Data Engineer in complex cloud environments.
• Strong hands-on expertise with AWS services (EC2, S3, Lambda, RDS, IAM, CloudWatch, etc.).
• Demonstrated experience with Airflow (Astronomer) setup, orchestration, and optimization.
• Proficiency in infrastructure as code (IaC) tools such as Terraform or CloudFormation.
• Experience with CI/CD pipelines and tools like Jenkins, GitHub Actions, or GitLab CI.
• Solid understanding of containerization technologies (Docker, Kubernetes).
• Working knowledge of Python and SQL for automation and data pipeline development.
• Familiarity with monitoring and observability tools (Grafana, Prometheus, CloudWatch).
• Strong grasp of data architecture principles and ETL design patterns.
• Financial services or regulated industry experience (desirable)