Airflow Architect – Big Data Systems at Austin TX

GigSumo.com; Austin (TX); On-site; USD 180,000 - 240,000; Full time

Job summary

GigSumo.com seeks an experienced Airflow Architect to design and lead enterprise-scale data orchestration solutions for modern big data platforms. You will apply deep expertise in Airflow, Spark, Hadoop, Kafka, and cloud-native architectures to build scalable, reliable data pipelines.

You will mentor teams, define DAG standards, and drive best practices for CI/CD and DevOps in AWS environments, ensuring high-quality product delivery and cost-efficient operations.

Qualifications

• 10+ years of experience in data engineering and big data platforms.
• Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
• Strong experience with workflow orchestration, metadata management, and pipeline monitoring.
• Hands-on ETL/ELT development and data pipeline automation.
• Experience with Kubernetes, Docker, and cloud-native deployments.

Responsibilities

• Architect and implement large-scale data orchestration frameworks using Apache Airflow.
• Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.
• Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
• Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
• Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
• Lead cloud-based data platform implementations on AWS.
• Establish CI/CD processes and DevOps practices for Airflow environments.
• Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
• Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.

Skills

Apache Airflow

Apache Spark

PySpark

Hadoop

Apache Kafka

Python

SQL

Shell scripting

Core Java

AWS

Tools

Databricks

AWS EMR

AWS S3

AWS Redshift

Kubernetes

Docker

Job description

About The Role

Need an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise in Apache Airflow, Big Data technologies, cloud-native architectures, and data engineering best practices to build scalable, reliable, and high-performance data pipelines.

Job Description

Need an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise in Apache Airflow, Big Data technologies, cloud-native architectures, and data engineering best practices to build scalable, reliable, and high-performance data pipelines.

We are seeking a detail-oriented and results-driven Airflow Architect. Strong domain knowledge in key practices including workflow orchestration, metadata management, pipeline monitoring, ETL, ELT and more is essential for this role. This role requires strong ability to collaborate with cross-functional teams, identify defects early in the development cycle, and ensure high-quality product delivery.

Job Title:

Airflow Architect

Experience Required:

10+ years

Primary Skills

• Apache Airflow
• Apache Spark
• PySpark
• Hadoop
• Apache Kafka
• Python
• SQL
• Shell scripting
• Core Java
• AWS

Secondary Skills

• workflow orchestration
• metadata management
• pipeline monitoring
• ETL
• ELT
• data pipeline automation
• Data Warehousing
• Data Lakes
• Lakehouse architectures
• NoSQL databases
• CI/CD processes
• DevOps practices

Soft Skills

• mentoring
• collaboration

Responsibilities

• Architect and implement large-scale data orchestration frameworks using Apache Airflow.
• Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.
• Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
• Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
• Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
• Lead cloud-based data platform implementations on AWS.
• Optimize pipeline performance, scalability, reliability, and cost efficiency.
• Establish CI/CD processes and DevOps practices for Airflow environments.
• Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
• Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.

Requirements

• Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
• Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
• Hands-on expertise in ETL/ELT development and data pipeline automation.
• Apache Spark / PySpark
• Hadoop ecosystem
• Apache Kafka
• Data Warehousing concepts
• Data Lakes and Lakehouse architectures
• NoSQL databases
• Python (advanced)
• SQL (advanced)
• Shell scripting
• Core Java (preferred)
• AWS (EMR, Glue, S3, Redshift)

Nice to Have

• 10+ years of experience in data engineering and big data platforms.
• 3+ years of architecture experience designing enterprise data solutions.
• Experience with Kubernetes, Docker, and cloud-native deployments.
• Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.
• Experience leading architecture reviews and technology strategy discussions.