AWS Data Engineer - AirFlow

Square One Resources; Manchester; Hybrid; GBP 80,000 - 100,000; Full time

Job summary

A leading financial services organization is seeking an Airflow/AWS Data Engineer to join their team. This role requires expertise in AWS services and Airflow orchestration, aimed at designing and maintaining cloud infrastructure and data pipelines. The position offers a contract rate of up to £400 per day and requires working 3 days a week in Manchester. Ideal candidates will have a strong background in data engineering and cloud environments.

Qualifications

• Experience as an AWS DevOps Engineer or Data Engineer in complex cloud environments.
• Hands-on expertise with AWS services like EC2, S3, and Lambda.
• Experience with Airflow (Astronomer) setup and orchestration.

Responsibilities

• Deploy and manage cloud infrastructure across Astronomer Airflow environments.
• Establish best practices for cloud security and performance.
• Develop ETL pipelines using AWS and SQL technologies.

Skills

AWS services expertise

Airflow orchestration

Terraform or CloudFormation

CI/CD pipelines

Python

SQL

Docker

Kubernetes

Tools

Grafana

Prometheus

Job description

Job Title: Airflow/AWS Data Engineer
Location: Manchester Area (3 days per week in the office)
Rate: Up to 400 per day inside IR35
Start Date: 03/11/2025
Contract Length: Until 31st December 2025
Job Type: Contract

Company Introduction:

An exciting opportunity has become available with one of our sector-leading financial services clients. They are seeking a talented AWS DevOps/Data Engineer to join their growing data engineering function. This role will play a key part in designing, deploying, and maintaining modern cloud infrastructure and data pipelines, with a focus on Airflow, AWS, and data platform automation.

Key Responsibilities:

• Deploy and manage cloud infrastructure across Astronomer Airflow and AccelData environments.
• Facilitate integration between vendor products and core systems, including data lakes, storage, and compute services.
• Establish and enforce best practices for cloud security, scalability, and performance.
• Configure and maintain vendor product deployments, ensuring reliability and optimized performance.
• Ensure high availability and fault tolerance for Airflow clusters.
• Implement and manage monitoring, alerting, and logging solutions for Airflow and related components.
• Perform upgrades, patches, and version management for platform components.
• Oversee capacity planning and resource optimization for Airflow workers and AWS resources.
• Manage integrations with source control systems (GitHub, GitLab) and CI/CD pipelines.
• Collaborate with AWS teams and internal stakeholders for pipeline scalability and optimization.
• Design and implement process improvements, including automation, data delivery optimization, and infrastructure re-design.
• Develop ETL pipelines and data workflows using AWS and SQL technologies.
• Partner with cross-functional teams (product, design, and leadership) to resolve technical issues and enhance platform capabilities.
• Build analytical tools and dashboards to leverage data pipelines for actionable business insights.

Key Requirements:

• Proven experience as an AWS DevOps Engineer or Data Engineer in complex cloud environments.
• Strong hands‑on expertise with AWS services (EC2, S3, Lambda, RDS, IAM, CloudWatch, etc.).
• Demonstrated experience with Airflow (Astronomer) setup, orchestration, and optimization.
• Proficiency in infrastructure as code (IaC) tools such as Terraform or CloudFormation.
• Experience with CI/CD pipelines and tools like Jenkins, GitHub Actions, or GitLab CI.
• Solid understanding of containerization technologies (Docker, Kubernetes).
• Working knowledge of Python and SQL for automation and data pipeline development.
• Familiarity with monitoring and observability tools (Grafana, Prometheus, CloudWatch).
• Strong grasp of data architecture principles and ETL design patterns.
• Financial services or regulated industry experience (desirable).