Airflow Expert

Red 11; United Kingdom; Remote; GBP 70,000 - 110,000; Full time

Job summary

Red 11 is seeking a Python-focused data engineering specialist in the UK to design scalable data pipelines and reusable code. You'll leverage Airflow, Pandas, and SQLAlchemy, with hands-on GCP experience (BigQuery, Cloud Storage) and GitHub workflows to ensure reliable, production-grade deployments.

Attention to debugging, scheduling, and observability with logging to GCS, plus collaboration across teams to optimize pipelines.

Qualifications

• Strong Python scripting for data engineering tasks.
• Expertise with Apache Airflow and DAG orchestration.
• Experience with GitHub workflows and CI/CD.
• Hands-on with Google Cloud Platform services.

Responsibilities

• Design and implement Python data pipelines for data engineering tasks.
• Build and maintain Airflow DAGs, operators, and sensors.
• Debug production issues and optimize scheduling and logging.
• Collaborate with teams using GCP services like BigQuery and Cloud Storage.

Skills

Python

Pandas

Requests

SQLAlchemy

Tools

Apache Airflow

GitHub

GCP

Job description

Python Scripting Expertise

Expert-level proficiency in Python (3.x) for data engineering, with strong experience in Scripting, pandas, requests, SQLAlchemy, custom libraries, and building reusable, production-grade code.

Apache Airflow

• Deep understanding of core concepts, including DAGs, tasks, operators, sensors, executors, hooks, and connections.
• Hands-on experience with Airflow architecture, scheduling, task dependencies, and extensibility through custom operators and plugins.
• Advanced configuration and debugging skills, including airflow.cfg and environment variables, logging setup (local and remote to GCS), and troubleshooting using dag.test(), CLI commands, log analysis, and task isolation.
• Proven ability to handle production issues such as scheduler delays, worker failures, resource constraints, and high-availability best practices.

GitHub

Proficiency with Git and GitHub workflows, including branching strategies (Git Flow/GitHub Flow), pull requests, code reviews, issue tracking, GitHub Actions for CI/CD, and collaborative development.

Google Cloud Platform (GCP)

Solid hands-on experience with GCP services, particularly Cloud Composer (managed Airflow), BigQuery (advanced SQL and Scripting), Cloud Storage, Dataflow, Pub/Sub, IAM, VPC, and Cloud Monitoring/Logging for pipeline observability.