Data Engineering Architect – Airflow & Azure

www.findapprenticeship.service.gov.uk - Jobboard; East Midlands; On-site; GBP 55,000 - 65,000; Full time

Job summary

A leading technology services provider is seeking a Data Engineering Architect specializing in Apache Airflow and Azure. This role involves designing scalable data workflows, enhancing existing systems, and collaborating with cross-functional teams to ensure data integrity. The ideal candidate will possess extensive knowledge of Airflow, Azure services, and CI/CD best practices. Strong problem-solving and communication skills are essential for success in this position.

Qualifications

• Proven experience as an Apache Airflow SME Architect or Developer in a production environment.
• Deep understanding of Airflow internals, including scheduler mechanics and executor types.
• Hands-on experience with Azure services, particularly Data Factory and Databricks.

Responsibilities

• Collaborate with stakeholders to gather CRM-related requirements for automated data workflows.
• Optimize existing Airflow environment and apply orchestration best practices.
• Build and maintain CI/CD pipelines for DAG deployment and testing.

Skills

Apache Airflow

Azure Data Factory

Python

CI/CD pipelines

Problem-solving

Monitoring tools (Prometheus, Grafana)

Tools

Azure DevOps

Docker

Kubernetes

Job description

Reference: DD252133_036

Vacancy: 1

Job title: Data Engineering Architect – Airflow & Azure

Location: Leicester

Salary: GBP 55000 to 65000 per annum

Published Date: 31 July 2025

Closing Date: 29 August 2025

Job Description (Main Duties and Responsibilities):
Collaborate with business stakeholders across Sales, Marketing, and Customer Support to gather CRM-related requirements and translate them into scalable, automated data workflows using Apache Airflow.
Assess existing CRM data workflows and Airflow DAGs for performance, reliability, and scalability. Recommend and implement enhancements to meet evolving business needs.
Work closely with data engineers, development teams, and CRM platform vendors (e.g., Dynamics 365, Salesforce) to integrate CRM systems with Airflow and Azure services (e.g., ADF, Databricks, Synapse).
Build Airflow jobs that support automated data extraction and transformation from CRM systems, enabling timely and actionable insights through analytics and reporting.
Implement and manage data quality checks and validation steps in DAGs to ensure the accuracy, consistency, and integrity of CRM data pipelines.
Provide documentation and user guidance for Airflow-based CRM workflows, and support business users in understanding orchestration processes and operational dashboards.
Design and maintain Airflow integrations between CRM systems and other enterprise platforms such as ERP, email marketing tools, and data lakes, enabling end-to-end visibility and data synchronization.

Key Skills, Qualifications and Experience Needed [The candidate must demonstrate these in all stages of assessment]
Proven experience as an Apache Airflow SME Architect or developer in a production environment.
Airflow Optimization & Best Practices: Analyze and enhance the existing Airflow environment, identifying and addressing performance bottlenecks. Apply orchestration best practices to improve scheduling and reliability.
DAG Development & Management: Design and implement scalable, modular, and reusable DAGs to support complex and evolving data workflows.
Azure Integration: Collaborate with cross-functional teams to integrate Airflow with Azure-native services such as Azure Data Factory, Azure Databricks, Azure Storage, and Azure Synapse.
CI/CD & Automation: Build and maintain CI/CD pipelines using Azure DevOps for DAG deployment, testing, and versioning. Automate workflows to support rapid and reliable delivery.
Monitoring & Operational Excellence: Establish standards for monitoring, alerting, and logging of Airflow jobs using tools like Prometheus, Grafana, and Azure Monitor to enable fast incident detection and resolution.
Architecture & Support: Provide hands-on support and architectural guidance for developing new data pipelines using Airflow and Azure technologies.
Documentation & Knowledge Sharing: Maintain detailed documentation for configurations, deployment processes, and operational procedures. Lead knowledge-sharing sessions and mentor junior team members.
Proven experience as an Apache Airflow SME or Lead Developer in a production environment.
Deep understanding of Airflow internals, including scheduler mechanics, executor types (Celery, Kubernetes), and plugin customization.
Hands-on experience with Azure services, particularly Data Factory, Databricks, Synapse, and Storage.
Proficient in Python, with a strong focus on modular, testable, and maintainable code.
Experience with Celery and KEDA for distributed task execution and autoscaling in data-centric or microservices architectures.
Skilled in building CI/CD pipelines using Azure DevOps (YAML, release pipelines, artifact management).
Familiar with containerization (Docker) and orchestration (Kubernetes) in the context of Airflow deployment.
Experience with observability tools such as Prometheus, Grafana, ELK Stack, or Azure Log Analytics.
Other Key skills:
Good analytical and Problem-solving skills
Good communication skills
A thorough approach and Self starter
Focus on quality and delivery
Working together in teams.
Leadership and effective decision making.
Flexible Attitude
Excellent customer service