Airflow Platform Engineer

Jobtailor; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Jobtailor in Hyderabad, India seeks an experienced Airflow Engineer to design, develop, and optimize enterprise Airflow DAGs. You will use PythonOperator, SQL operators, API/HTTP operators, and KubernetesPodOperator to build reliable workflows and enhance data pipelines.

The role involves debugging, CI/CD integration, and azure/cloud interactions. Responsibilities include templating best practices, onboarding others to self-service workflows, and ensuring platform stability with monitoring and

Qualifications

• 4–6 years of experience in building and maintaining production Airflow workloads.
• Proven experience designing DAGs with PythonOperator, SQL operators, API/HTTP operators, and KubernetesPodOperator.
• Strong Python development skills for production-grade workflows and supporting libraries.
• Hands-on Kubernetes (AKS preferred) to support Airflow runtime.
• Experience with DBT and modern data pipelines.
• Familiarity with CI/CD tools (Git, Jenkins, Docker).
• Exposure to Azure or other cloud services.

Responsibilities

• Design, develop, and optimize Airflow DAGs for enterprise use cases.
• Demonstrate expertise with Airflow operators and orchestration patterns.
• Apply advanced DAG patterns including dynamic DAGs, branching, sensors, retries, and error handling.
• Troubleshoot DAG failures, performance issues, and integration challenges.
• Enable orchestration of modern data workflows including DBT pipelines.
• Provide reusable templates, best practices, and onboarding support for self-service adoption.
• Support Airflow platform deployed on AKS, including scheduler, webserver, workers, and metadata services.
• Troubleshoot issues related to task execution, scheduling delays, and resource bottlenecks.
• Diagnose Kubernetes-level issues involving pods, networking, storage, and RBAC.
• Work with Azure services such as Key Vault, Storage, and networking.
• Support CI/CD pipelines for DAG and container-based deployments.
• Improve platform stability, monitoring, and alerting.

Skills

Apache Airflow

Python

SQL

Kubernetes

DBT

CI/CD

Azure

Linux

Tools

Git

Jenkins

Docker

Job description

Responsibilities

• Design, develop, and optimize Airflow DAGs for enterprise use cases
• Demonstrate strong experience with operators such as PythonOperator, SQL operators, API/HTTP operators, KubernetesPodOperator etc.
• Apply advanced DAG patterns including dynamic DAGs, branching, sensors, retries, and error handling
• Troubleshoot DAG failures, performance issues, and integration challenges
• Enable orchestration of modern data workflows including DBT pipelines
• Provide reusable templates, best practices, and onboarding support to drive self‑service adoption
• Support Airflow platform deployed on AKS, covering scheduler, webserver/API, workers, and metadata services
• Troubleshoot issues related to task execution, scheduling delays, and resource bottlenecks
• Diagnose Kubernetes-level issues involving pods, networking, storage, and RBAC
• Work with Azure services such as Key Vault, Storage, and networking
• Support CI/CD pipelines for DAG and container-based deployments
• Improve platform stability, monitoring, and alerting

Requirements

• 4–6 years of experience
• Strong experience with Apache Airflow (3.x or later) in production environments
• Proven experience in DAG development using operators and orchestration patterns
• Working knowledge of Kubernetes (AKS preferred) to support Airflow runtime
• Expert‑level Python knowledge for building, debugging, and maintaining production‑grade workflows and supporting libraries
• Expertise in writing unit, integration, and DAG validation tests for Airflow workflows, including mocking operators, validating dependencies, and ensuring reliability in CI/CD pipelines
• Experience with DBT and modern data pipelines
• Experience with CI/CD tools (Git, Jenkins, Docker)
• Familiarity with Azure or other cloud platforms