Astronomer Architect-6

Realign Llc; Plano (TX); On-site; USD 140,000 - 210,000; Full time

Job summary

Realign Llc in Plano, TX is seeking an Astronomer Architect to lead production Airflow on Kubernetes, implement scalable data pipelines, and drive platform reliability. You will own the control plane, orchestration, and observability, while mentoring engineers and aligning with security standards.

The role requires 5–8+ years building data/platform systems and 3+ years running Airflow at scale, with deep Kubernetes expertise and strong Python skills.

Qualifications

• 5-8+ years building/operating data or platform systems.
• 3+ years running Airflow in production at scale.
• Strong Kubernetes experience with Helm and scaling policies.
• Experience with Terraform modules and GitOps CI/CD.
• Proficiency in Python for operator development and scripting.

Responsibilities

• Own the Airflow control plane and developer experience end-to-end, including security and observability.
• Build automation infrastructure and partner with data/engineering teams for scalable pipelines.
• Architect and operate production-grade Airflow on Kubernetes with HA, capacity planning, and performance tuning.
• Standardize developer experience with templates, libraries, and secrets management.
• Implement comprehensive observability, dashboards, tracing, and alerting.
• Lead incident responses and develop runbooks and platform roadmaps.

Skills

Airflow in production

Kubernetes

Python

Terraform

GitOps

Observability

Incident leadership

CI/CD workflows

Tools

Helm

Argo CD/Flux

Snowflake/BigQuery/Redshift

Go/Java familiarity

Job description

Plano, Texas 75023 Posted October 19th, 2026

Job Type: Full Time

Job Category: IT

Job Description

Role: Astronomer Architect

Location: Plano, TX – Onsite

Fulltime – Permanent

Job Description

Must Have Technical/Functional Skills

• Architect, deploy, and operate production-grade Airflow on Kubernetes including all components and user application dependencies, with focus on upgrades, capacity planning, HA, security, and performance tuning
• Operate a multi-scheduler ecosystem: determine when to use Airflow, distributed compute schedulers, or lightweight task runners based on workload requirements; provide unified developer experience across schedulers
• Build automation infrastructure: Terraform modules and Helm charts with GitOps-driven CI/CD for environment provisioning, upgrades, and zero-downtime rollouts
• Standardize the developer experience: DAG repo templates, shared operator libraries, connection and secrets management, dependency packaging, code ownership, linting, unit testing, and pre-commit hooks
• Implement comprehensive observability: metrics collection, dashboards, distributed tracing, SLA/latency monitoring, intelligent alerting, and runbook automation
• Enable resilient workflow patterns: build idempotency frameworks, retry/backoff strategies, deferrable operators and sensors, dynamic task mapping, and data-aware scheduling
• Ensure reliability at enterprise scale: architect and tune resource allocation (pools, queues, concurrency limits) to support high-throughput workloads; optimize large-scale backfill strategies; develop comprehensive runbooks and lead incident response/postmortems
• Partner with teams across the organization to provide enablement, documentation, and self-service tooling
• Mentor engineers, contribute to platform roadmap and technical standards, and drive engineering best practices
• 5-8+ years building/operating data or platform systems; 3+ years running Airflow in production at scale (hundreds-thousands of DAGs and high task throughput).
• Deep Airflow expertise: DAG design and testing, idempotency, deferrable operators/sensors, dynamic task mapping, task groups, datasets, pools/queues, SLAs, retries/backfills, cross DAG dependencies
• Strong Kubernetes experience running Airflow and supporting services: Helm, autoscaling, node/pod tuning, topology spread, network policies, PDBs, and blue/green or canary strategies.
• Automation-first mindset: Terraform, Helm, GitOps (Argo CD/Flux), and C/CD for platform lifecycle; policy as code (ON/Gatekeeper/Conftest) for DAG, connection, and secrets changes.
• Proficiency in Python for authoring operators/hooks/utilities; solid Bash; familiarity with Go or Java is a plus.
• Observability and SRE practices: Prometheus/Grafana/StatsD, centralized logging, alert design, capacity/throughput modeling, performance tuning.
• Data platform experience with at least one major cloud (AWS/Azure/GCP) and systems like Snowflake/BigQuery/Redshift, Databricks/Spark, EMR/Dataproc; strong grasp of IAM, VPC networking, and storage (S3/GCS/ADLS).
• Security/compliance: SS0/OIDC, RBAC, secrets management (Vault/Secrets Manager), auditing, least privilege connection management, and change control.
• Proven incident leadership, runbook creation, and platform roadmap execution; excellent cross-functional communication.

Roles & Responsibilities

• You'll own the Airflow control plane and developer experience end-to-end-architecture, automation, security, observability, and reliability-while also evaluating and operating complementary schedulers where appropriate
• You'll build automation infrastructure and partner across data, trading, and engineering teams to deliver mission-critical pipelines at scale
• Architect, deploy, and operate production grade Airflow on Kubernetes including all components and user application dependencies, with focus on upgrades, capacity planning, HA, security, and performance tuning Operate a multi-scheduler ecosystem: determine when to use Airflow, distributed compute schedulers, or lightweight task runners based on workload requirements; provide unified developer experience across schedule Build automation infrastructure: Terraform modules and Helm charts with GitOps-driven CI/CD for environment provisioning, upgrades, and zero-downtime rollouts
• Standardize the developer experience: DAG repo templates, shared operator libraries, connection and secrets management, dependency packaging, code ownership, linting, unit testing, and pre-commit hooks Implement comprehensive observability: metrics collection, dashboards, distributed tracing, SLA/latency monitoring, intelligent alerting, and runbook automation
• Enable resilient workflow patterns: build idempotency frameworks, retry/backoff strategies, deferrable operators and sensors, dynamic task mapping, and data-aware scheduling
• Ensure reliability at enterprise scale: architect and tune resource allocation (pools, queues, concurrency limits) to support high throughput workloads; optimize large-scale backfill strategies; develop comprehensive runbooks and lead incident response/postmortems
• Partner with teams across the organization to provide enablement, documentation, and self service tooling
• Mentor Engineers, contribute to platform roadmap and technical standards, and drive engineering best practices

Nice To Have :-

• Experience leading migrations to/from Airflow
• Open Lineage/Marquez adoption; Great Expectations or other date quality frameworks; data contracts.
• dbt Core/Cloud orchestration patterns (state management, artifacts, slim Ci).
• Cost optimization and capacity planning for schedulers and workers; spot instance strategies.
• Multi-region HA/DR for Airflow metadata DB; backup/restore and disaster drills.
• Building internal developer platforms/portals (e.g., Backstage) for self-service pipelines.
• Contributions to Apache Airflow or provider packages; familiarity with recent AlPs/Airflow 2.7+ features