Airflow Architect – Big Data

Programmers.io; Austin (TX); On-site; USD 150,000 - 190,000; Full time

Job summary

Programmers.io is seeking an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. You will leverage deep expertise in Apache Airflow, big data technologies, cloud-native architectures, and data engineering best practices to build scalable, reliable data pipelines.

The role focuses on designing large-scale orchestration frameworks, developing ETL/ELT pipelines, and leading cloud deployments on AWS while mentoring

Qualifications

• Airflow & Data Engineering
• Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
• Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
• Hands-on expertise in ETL/ELT development and data pipeline automation.
• Hadoop ecosystem
• Data Warehousing concepts
• Data Lakes and Lakehouse architectures

Responsibilities

• Architect and implement large-scale data orchestration frameworks using Apache Airflow.
• Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.
• Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
• Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
• Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
• Lead cloud-based data platform implementations on AWS
• Optimize pipeline performance, scalability, reliability, and cost efficiency.
• Establish CI/CD processes and DevOps practices for Airflow environments.
• Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
• Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.

Skills

Airflow DAG design

Workflow orchestration

ETL/ELT development

Hadoop ecosystem

Data Warehousing concepts

Data Lakes/Lakehouse architectures

Kubernetes

Docker

Cloud-native deployments

Data engineering best practices

Tools

Apache Spark

Kafka

Databricks

Kubernetes

Docker

Job description

Need an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise in Apache Airflow, Big Data technologies, cloud-native architectures, and data engineering best practices to build scalable, reliable, and high-performance data pipelines.

Key Responsibilities

• Architect and implement large-scale data orchestration frameworks using Apache Airflow.
• Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.
• Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
• Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
• Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
• Lead cloud-based data platform implementations on AWS
• Optimize pipeline performance, scalability, reliability, and cost efficiency.
• Establish CI/CD processes and DevOps practices for Airflow environments.
• Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
• Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.

Required Skills

• Airflow & Data Engineering
• Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
• Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
• Hands-on expertise in ETL/ELT development and data pipeline automation.
• Hadoop ecosystem
• Data Warehousing concepts
• Data Lakes and Lakehouse architectures

Preferred Qualifications

• 10+ years of experience in data engineering and big data platforms.
• 3+ years of architecture experience designing enterprise data solutions.
• Experience with Kubernetes, Docker, and cloud-native deployments.
• Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.
• Experience leading architecture reviews and technology strategy discussions.

Success Metrics

• Pipeline reliability and SLA adherence
• Workflow performance optimization