Astronomer Architect

Tata Consultancy Services Limited; Plano (TX); On-site; USD 100,000 - 120,000; Full time

Benefits offered by this job

Discretionary Annual Incentive.

Comprehensive Medical Coverage: Health

Parental Leaves

Job summary

Tata Consultancy Services Limited is seeking an experienced Technical Architect to design, operate, and scale our workflow orchestration platform centered on Apache Airflow. You will own the Airflow control plane, developer experience, and security, while evaluating complementary schedulers and building automation across data teams in Plano, TX.

Key responsibilities include operating Airflow on Kubernetes, driving GitOps-based CI/CD for environments, and mentoring engineers to advance platform

Qualifications

• 5-8+ years building/operating data or platform systems.
• 3+ years running Airflow in production at scale.

Responsibilities

• Own the Airflow control plane and developer experience end-to-end—architecture, automation, security, observability, and reliability.
• Build automation infrastructure and partner across data, trading, and engineering teams to deliver mission-critical pipelines at scale.
• Architect, deploy, and operate production-grade Airflow on Kubernetes with upgrades, capacity planning, HA, security, and performance tuning.
• Operate a multi-scheduler ecosystem and provide a unified developer experience across schedulers.
• Implement comprehensive observability including metrics, dashboards, tracing, SLA/latency monitoring, alerting, and runbook automation.
• Lead incident response, runbooks, and enterprise-scale reliability improvements.
• Mentor engineers and contribute to platform roadmap and standards.

Skills

AWS

Azure

Education

Bachelor of Computer Science

Job description

Job Description

Must Have Technical/Functional Skills

We're seeking a seasoned engineer to design, operate, and scale our workflow orchestration platform with a primary focus on Apache Airflow. You'll own the Airflow control plane and developer experience end-to- end--architecture, auto security, observability, and reliability-while also evaluating and operating complementary schedulers where appropriate. You'll build automation infrastructure and partner across data, trading, and engineering teams to deliver minion-critica pipelines at scale.

Architect, deploy, and operate production-grade Airflow on Kubernetes including all components and user application dependencies, with focus on upgrades, capacity planning, HA, security, and performance tuning.

• Operate a multi-scheduler ecosystem: determine when to use Airflow, distributed compute schedulers, or lightweight task runners based on workload requirements; provide unified developer experience across schedulers
• Build automation infrastructure: Terraform modules and Helm charts with GitOps-driven CI/CD for environment provisioning, upgrades, and zero-downtime rollouts
• Standardize the developer experience: DAG repo templates, shared operator libraries, connection and secrets management, dependency packaging, code ownership, linting, unit testing, and pre-commit hooks
• Implement comprehensive observability: metrics collection, dashboards, distributed tracing, SLA/latency monitoring, intelligent alerting, and runbook automation
• Enable resilient workflow patterns: build idempotency frameworks, retry/backoff strategies, deferrable operators and sensors, dynamic task mapping, and data-aware scheduling
• Ensure reliability at enterprise scale: architect and tune resource allocation (pools, queues, concurrency limits) to support high-throughput workloads; optimize large-scale backfill strategies; develop comprehensive runbooks and lead incident response/postmortems
• Partner with teams across the organization to provide enablement, documentation, and self-service tooling
• Mentor engineers, contribute to platform roadmap and technical standards, and drive engineering best practices
• 5-8+ years building/operating data or platform systems; 3+ years running Airflow in production at scale (hundreds-thousands of DAGs and high task throughput).
• Deep Airflow expertise: DAG design and testing, idempotency, deferrable operators/sensors, dynamic task mapping, task groups, datasets, pools/queues, SLAs, retries/backfills, cross DAG dependencies
• Strong Kubernetes experience running Airflow and supporting services: Helm, autoscaling, node/pod tuning, topology spread, network policies, PDBs, and blue/green or canary strategies.
• Automation-first mindset: Terraform, Helm, GitOps (Argo CD/Flux), and C/CD for platform lifecycle; policy as code (ON/Gatekeeper/Conftest) for DAG, connection, and secrets changes.
• Proficiency in Python for authoring operators/hooks/utilities; solid Bash; familiarity with Go or Java is a plus.
• Observability and SRE practices: Prometheus/Grafana/StatsD, centralized logging, alert design, capacity/throughput modeling, performance tuning.
• Data platform experience with at least one major cloud (AWS/Azure/GCP) and systems like Snowflake/BigQuery/Redshift, Databricks/Spark, EMR/Dataproc; strong grasp of IAM, VPC networking, and storage (S3/GCS/ADLS).
• Security/compliance: SS0/OIDC, RBAC, secrets management (Vault/Secrets Ma nager), auditing, least privilege connection management, and change control.
• Proven incident leadership, runbook creation, and platform roadmap execution; excellent cross-functional communication.

Roles & Responsibilities

• You’ll own the Airflow control plane and developer experience end-to-end-architecture, automation, security, observability, and reliability-while also evaluating and operating complementary schedulers where appropriate
• You’ll build automation infrastructure and partner across data, trading, and engineering teams to deliver mission-critical pipelines at scale
• Architect, deploy, and operate production grade Airflow on Kubernetes including all components and user application dependencies, with focus on upgrades, capacity planning, HA, security, and performance tuning Operate a multi-scheduler ecosystem: determine when to use Airflow, distributed compute schedulers, or lightweight task runners based on workload requirements; provide unified developer experience across schedule Build automation infrastructure: Terraform modules and Helm charts with GitOps-driven CI/CD for environment provisioning, upgrades, and zero-downtime rollouts
• Standardize the developer experience: DAG repo templates, shared operator libraries, connection and secrets management, dependency packaging, code ownership, linting, unit testing, and pre-commit hooks Implement comprehensive observability: metrics collection, dashboards, distributed tracing, SLA/latency monitoring, intelligent alerting, and runbook automation
• Enable resilient workflow patterns: build idempotency frameworks, retry/backoff strategies, deferrable operators and sensors, dynamic task mapping, and data-aware scheduling
• Ensure reliability at enterprise scale: architect and tune resource allocation (pools, queues, concurrency limits) to support high throughput workloads; optimize large-scale backfill strategies; develop comprehensive runbooks and lead incident response/postmortems
• Partner with teams across the organization to provide enablement, documentation, and self service tooling
• Mentor Engineers, contribute to platform roadmap and technical standards, and drive engineering best practices

Nice To Have :-

• Experience leading migrations to/from Airflow
• Open Lineage/Marquez adoption; Great Expectations or other date quality frameworks; data contracts.
• dbt Core/Cloud orchestration patterns (state management, artifacts, slim Ci).
• Cost optimization and capacity planning for schedulers and workers; spot instance strategies.
• Multi-region HA/DR for Airflow metadata DB; backup/restore and disaster drills.
• Building internal developer platforms/portals (e.g., Backstage) for self-service pipelines.
• Contributions to Apache Airflow or provider packages; familiarity with recent AlPs/Airflow 2.7+ features

TCS Employee Benefits Summary:

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sic k Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Salary Range: $100,000-$120,000 a year

Location

Plano, TX

Function

TECHNOLOGY

Role

Technical Architect

ID

435537

Skills

AWS | Azure

Salary Range

$100,000 - $120,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE