Senior Airflow/ Cloud Engineer (Workload Automation)

Deutsche Bank; Birmingham; On-site; GBP 65,000 - 95,000; Full time

Job summary

Deutsche Bank in the United Kingdom is seeking a software engineer focused on workload automation to ensure availability, latency, and efficiency of centralized platforms such as Control-M, Automic, Airflow and Google Cloud Composer.

You will develop in Python, integrate REST APIs, and design automation using Ansible and Terraform, contributing to monitoring, change management and capacity planning across batch scheduling systems.

Qualifications

• Experience with Python development and API orchestration for workload automation platforms.
• Proficiency in Airflow and Google Cloud Platform services including Cloud Composer, BigQuery, Dataflow, Dataproc.
• Hands-on experience with Control-M or Automic is a plus and REST-based API orchestration.

Responsibilities

• Develop and implement automation for Control-M, Automic, Airflow and Google Cloud Composer.
• Integrate with REST APIs and orchestration layers to drive workload automation.
• Design improvements and replacements for existing processes, especially replacing Control-M/Automic with Airflow or Cloud Composer.
• Install, configure, monitor Airflow clusters, tune DAGs and data pipelines.
• Contribute to capacity planning, change management and proactive monitoring across centralized platforms.

Skills

Python

API integration

Orchestration

REST API

Airflow

Google Cloud Platform

Control-M

Automic

Ansible YAML

SDLC / Agile

DAG development

Performance tuning

Data engineering basics

Tools

Terraform

Cloud Composer

BigQuery

Dataflow

Cloud Functions

Dataproc

Postgres

Ansible

Job description

• This software engineering position is specifically focused within Workload Automation and is responsible for availability, latency, performance, efficiency, change management, monitoring, automation, emergency response, and capacity planning for the centralized platforms of Control-M, Automic, Airflow as well as Google services Cloud Composer, Workflows and Scheduler which are utilized across the Bank for workload automation/job scheduling/batch job management
• This position will complement someone with a software engineering mindset, who is proficient coding with Python, has experience with API integration and orchestration, ServiceNOW integration and Ansible yaml development to proactively build and implement services for Control-M, Automic and Google Cloud Composer automation
• Demonstrate a wide range of technological expertise including Airflow, Google Cloud Platform, Control-M or Automic and a general understanding of industry practice around REST based API orchestration
• Be an active, contributing engineer for the integration of Apache Airflow into the IT landscape by technically and organisationally helping to drive the adoption
• Design and implementation of new concepts for improvement or replacement of existing processes, especially from Control-M or Automic towards Airflow or Google Could Composer
• Integration, configuration, upgrade, maintenance and introduction of new capabilities to our critical Workload Automation products responsible for batch scheduling and API orchestration
• Being able to design concepts and take over development activities, identify automation opportunities as well as solutions in a self-reliant manner
• Sound and relevant Airflow or Workload Automation architecture experience is a "must"

Experience with Workload automation tooling, preferably Apache Airflow, Control-M and/or Google Cloud Composer using standard agile SDLC techniquesHands on experience in Google Cloud Platform using following GCP services Terraform, Cloud Composer, BigQuery, Dataflow, Cloud Functions, Dataproc, Google Transfer Service, PostgresExperience with installing, configuring and monitoring Airflow cluster, strong PythonExperience in developing guidelines for Airflow clusters and DAG’s, working knowledge of performance tuning of the DAG’s and task implementation, developing DAG - data pipeline to on-board and change management of datasetsUnderstanding of Airflow rest services and integration of Airflow ON A platform eco-system