Senior Apache Airflow Platform Engineer

Veriipro; Plano (TX); Sur place; USD 180 000 - 240 000; Plein temps

Résumé du poste

Veriipro is seeking a senior data/platform engineer to architect and operate production Airflow/Astronomer platforms on Kubernetes. You will drive upgrades, capacity planning, security, and performance while automating provisioning with Terraform, Helm, and GitOps.

You will develop reusable DAGs, operators, and self-service tools to optimize workflows. You will implement thorough monitoring, incident response, and disaster recovery, and collaborate with data engineering teams to set platform

Qualifications

• 6+ years of experience in data engineering, platform engineering, or infrastructure systems.
• 3+ years of hands-on Apache Airflow production experience at scale.
• Strong expertise in Astronomer, Airflow DAGs, operators, sensors, dynamic task mapping, scheduling, and workflow optimization.
• Strong Kubernetes, Helm, Terraform, GitOps (Argo CD/Flux), and CI/CD experience.
• Proficiency in Python and Bash scripting.
• Experience with Prometheus, Grafana, centralized logging, monitoring, and SRE practices.
• Knowledge of AWS, Azure, or GCP, including IAM, networking, and cloud storage.
• Experience with security, RBAC, SSO/OIDC, secrets management, incident response, and high-availability architecture.

Responsabilités

• Architect, deploy, and manage production-grade Apache Airflow/Astronomer platforms on Kubernetes.
• Manage platform upgrades, high availability, scalability, capacity planning, security, and performance optimization.
• Automate infrastructure provisioning and platform lifecycle using Terraform, Helm, GitOps, and CI/CD pipelines.
• Develop reusable DAG templates, custom operators, shared libraries, testing frameworks, and self-service tools.
• Implement monitoring, logging, alerting, SLA tracking, incident management, and disaster recovery.
• Optimize workflow scheduling, task concurrency, retries, backfills, and resource utilization.
• Implement access controls, secrets management, audit logging, and security best practices.
• Collaborate with data engineering and platform teams, mentor engineers, and define platform standards and roadmaps.

Connaissances

Data engineering experience

Airflow at scale

Astronomer

Airflow DAGs

Operators

Sensors

Dynamic task mapping

Scheduling

Workflow optimization

Kubernetes

Helm

Terraform

GitOps (Argo CD/Flux)

CI/CD

Python

Bash scripting

Prometheus

Grafana

Logging/Monitoring

SRE practices

AWS

Azure

GCP

IAM

Networking

Cloud storage

Security

RBAC

SSO/OIDC

Secrets management

Incident response

HA architecture

Snowflake

BigQuery

Redshift

Databricks

Spark

dbt

OpenLineage

Marquez

Data quality frameworks

Data contracts

Disaster recovery

Platform migrations

Cost optimization

Airflow contributions

Internal developer platforms

Outils

Kubernetes

Argo CD

Flux

Description du poste

Roles & Responsibilities

• Architect, deploy, and manage production-grade Apache Airflow/Astronomer platforms on Kubernetes.

• Manage platform upgrades, high availability, scalability, capacity planning, security, and performance optimization.

• Automate infrastructure provisioning and platform lifecycle using Terraform, Helm, GitOps, and CI/CD pipelines.

• Develop reusable DAG templates, custom operators, shared libraries, testing frameworks, and self-service tools.

• Implement monitoring, logging, alerting, SLA tracking, incident management, and disaster recovery.

• Optimize workflow scheduling, task concurrency, retries, backfills, and resource utilization.

• Implement access controls, secrets management, audit logging, and security best practices.

• Collaborate with data engineering and platform teams, mentor engineers, and define platform standards and roadmaps.

Required Skills

• 6+ years of experience in data engineering, platform engineering, or infrastructure systems.

• 3+ years of hands-on Apache Airflow production experience at scale.

• Strong expertise in Astronomer, Airflow DAGs, operators, sensors, dynamic task mapping, scheduling, and workflow optimization.

• Strong Kubernetes, Helm, Terraform, GitOps (Argo CD/Flux), and CI/CD experience.

• Proficiency in Python and Bash scripting.

• Experience with Prometheus, Grafana, centralized logging, monitoring, and SRE practices.

• Knowledge of AWS, Azure, or GCP, including IAM, networking, and cloud storage.

• Experience with security, RBAC, SSO/OIDC, secrets management, incident response, and high-availability architecture.

Nice to Have

• Experience with Snowflake, BigQuery, Redshift, Databricks, Spark, or dbt.

• Familiarity with OpenLineage, Marquez, data quality frameworks, and data contracts.

• Experience with multi-region disaster recovery, platform migrations, and cost optimization.

• Contributions to Apache Airflow or development of internal developer platforms.