Senior Data Engineer (Airflow Specialist)

Hvantage Technologies Inc USA; India; Hybrid; INR 1,500,000 - 2,000,000; Full time

Job summary

Hvantage Tech Solutions Pvt. Ltd. is looking for a Senior Data Engineer (Airflow Specialist) to design and implement an enterprise data orchestration platform. The role requires expertise in Apache Airflow, Python, and cloud platforms like Azure.

Ideal candidates will have over 5 years of experience in Data Engineering and a strong focus on building scalable data infrastructures. This position offers remote and hybrid working options.

Qualifications

• 5+ years in Data Engineering, with 2-3 years focused on scaling Airflow.
• Advanced proficiency in Python and expert-level SQL.
• Experience migrating legacy schedulers to Airflow.

Responsibilities

• Lead design and implementation of the enterprise data orchestration platform.
• Author complex, modular, and idempotent DAGs using Python.
• Integrate Airflow with heavy-compute environments like Azure Databricks.

Skills

Apache Airflow

Python

SQL

Azure Services

GitHub Actions

CI/CD

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Terraform

Prometheus

Grafana

Job description

Job title: Senior Data Engineer (Airflow Specialist)

Company: Hvantage Tech Solutions Pvt. Ltd.

Department: Enterprise Applications / Healthcare IT

Location: Remote / Hybrid / Onsite (as applicable)

Employment Type: Full-Time

Experience Required: 5+ Years

Work Mode: Must be comfortable working with global teams and US clients

Methodology: SAFe Agile (Mandatory)

About Hvantage Tech Solutions Pvt. Ltd

Hvantage Tech Solutions Pvt. Ltd. is a global technology and healthcare IT services provider delivering advanced digital solutions for healthcare payers, providers, and life sciences organizations. Our expertise spans enterprise platforms, healthcare interoperability, AI‑enabled analytics, cloud‑native architectures, and digital member engagement platforms.

Role Overview

We are seeking a versatile Senior Data Engineer to lead the design, implementation, and scaling of our enterprise data orchestration platform using Apache Airflow. This role requires a “builder” mindset, combining the agility to create robust pipelines from the ground up with the maturity to ensure they are scalable, secure, and integrated into a complex cloud ecosystem. As the primary architect for workflow automation, you will move beyond simple task scheduling to build a resilient, self‑healing data infrastructure.

Key Responsibilities

• Data Orchestration & Pipeline Engineering: Author complex, modular, and idempotent DAGs using Python and the Airflow TaskFlow API.
• API/Framework Development: Build custom operators, hooks, and sensors to standardize data integrations across the organization.
• High‑Compute Integration: Integrate Airflow with heavy‑compute environments including Azure Databricks, Spark clusters, and cloud data warehouses.
• Platform Management: Optimize Airflow executors (Celery or Kubernetes) to manage high‑concurrency workloads and tune backend databases (PostgreSQL/MySQL).
• Security & Compliance: Implement enterprise‑grade security protocols, including role‑based access control (RBAC), secret management (Azure Key Vault/HashiCorp Vault), and OAuth integration.
• DevOps & Reliability: Build and maintain automated CI/CD deployment pipelines for DAGs and infrastructure‑as‑code using Terraform or Pulumi.
• Observability: Implement advanced monitoring and alerting using Prometheus, Grafana, or the ELK stack to track pipeline health and SLA breaches.
• Leadership & Mentorship: Establish data engineering best practices across the team and conduct rigorous code reviews to ensure architectural integrity.

Technical Requirements

• Orchestration: Expert knowledge of Apache Airflow 3.x+ (XComs, providers, and dynamic task mapping).
• Languages: Advanced proficiency in Python and expert‑level SQL.
• Cloud Platforms: Proficiency in Azure services (Data Factory, Blob Storage, AKS) or equivalent in AWS or GCP.
• DevOps: Hands‑on experience with CI/CD tools such as GitHub Actions, Azure DevOps, or Jenkins.

Qualifications

• Experience: 5+ years in Data Engineering, with a minimum of 2‑3 years focused specifically on scaling Airflow in production environments.
• Education: Bachelor’s or Master’s degree in Computer Science, Information Systems, or a related field.
• Skills: Experience migrating legacy schedulers to Airflow; background in software development (Java or .NET); knowledge of data quality frameworks such as Great Expectations or dbt.

Competencies

• Strong focus on long‑term maintenance and architectural integrity.
• Ability to move quickly in an agile environment without compromising on code quality.
• Exceptional problem‑solving skills for debugging complex data orchestration issues.