Airflow Developer

Genpact Ltd.

Hyderabad, Bengaluru, Delhi

On-site

INR 900,000 - 1,500,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Genpact Ltd. in Hyderabad, India, seeks an Airflow Developer / Data Engineer with deep expertise in Azure Databricks to design and orchestrate next-generation data pipelines. You will develop scalable ETL/ELT workflows, manage complex DAGs in Airflow, and optimize big data processing within Azure.

Responsibilities include DAG development, custom operators, performance tuning, and robust monitoring. Proficiency in PySpark, Delta Lake, and secure Azure integrations is essential for success.

Qualifications

  • 3+ years of hands-on experience with Apache Airflow or similar workflow orchestration tools.
  • Experience building data pipelines in Azure Databricks with PySpark and Delta Lake.
  • Strong programming skills in Python and SQL.
  • Familiarity with Azure services (ADLS Gen2, Key Vault) and DevOps practices.

Responsibilities

  • Design, write, and maintain complex Apache Airflow DAGs in enterprise data pipelines.
  • Develop custom Airflow operators, hooks, and sensors for integration with systems.
  • Tune performance of Airflow tasks to optimize scheduling and reduce costs.
  • Implement robust monitoring, logging, and alerting for pipelines (Slack, email, PagerDuty).
  • Build scalable ETL/ELT workflows and manage data lakehouse architectures in the Azure ecosystem.

Skills

Airflow
Azure Databricks
Python
SQL
PySpark
Data modeling

Tools

Airflow
Azure Databricks
Delta Lake
Spark
Azure DevOps

Job description

Role & responsibilities

Airflow Developer / Data Engineer with deep expertise in Azure Databricks to design, build, and orchestrate our next-generation data pipelines. In this role, you will be responsible for developing scalable ETL/ELT workflows, managing complex DAGs (Directed Acyclic Graphs) in Apache Airflow, and optimizing big data processing workloads within the Azure ecosystem.

Workflow Orchestration & Airflow Management
  • DAG Development: Design, write, and maintain complex Apache Airflow DAGs using clean, reusable, and modular Python code.
  • Custom Operators: Build and implement custom Airflow operators, hooks, and sensors to integrate with proprietary or niche systems.
  • Performance Tuning: Optimize Airflow scheduling, concurrency limits, and task execution to prevent bottlenecks and reduce infrastructure costs.
  • Monitoring & Alerting: Implement robust logging, error handling, and alerting mechanisms (e.g., Slack, PagerDuty, email notifications) for pipeline failures.
Data Engineering & Azure Databricks
  • Pipeline Development: Design and deploy scalable data pipelines using PySpark / Delta Lake within Azure Databricks.
  • Databricks Integration: Use Airflows DatabricksSubmitRunOperator or DatabricksRunNowOperator to orchestrate Databricks Notebooks and multi-task workflows smoothly.
  • Performance Optimization: Optimize Spark jobs by configuring cluster sizes, addressing data skew, caching efficiently, and managing file sizing (e.g., Z-Ordering, Optimize).
  • Data Lakehouse Architecture: Maintain Medallion Architecture (Bronze, Silver, Gold layers) inside Azure Data Lake Storage (ADLS Gen2).
Cloud Infrastructure & DevOps
  • CI/CD Automation: Build and automate deployment pipelines for Airflow DAGs and Spark code using Azure DevOps or GitHub Actions.
  • Access & Security: Configure secure connections between Airflow and Azure services using Azure Key Vault, Managed Identities, and Service Principals.
  • Environment Maintenance: Collaborate with platform teams to manage Airflow environments (e.g., Astronomer, MWAA, Docker/Kubernetes containerized setups).
Preferred candidate profile
Technical Skills
  • Apache Airflow: 3+ years of hands-on experience orchestrating enterprise-grade data pipelines. Deep knowledge of Airflow 2.x features (TaskFlow API, Dynamic Task Mapping).
  • Azure Databricks & Spark: 3+ years of experience processing large-scale datasets with PySpark, Spark SQL, and Delta Lake.
  • Programming: Advanced proficiency in Python and SQL.
  • Azure Cloud Platform: Solid familiarity with Azure Data Lake Storage (ADLS Gen2), Azure Key Vault, and Azure DevOps.
  • Data Modeling: Strong understanding of data warehousing concepts, star/snowflake schemas, and slow-changing dimensions (SCD).
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Airflow Platform Engineer
Airflow Platform Engineer

Jade Global • Hyderabad

On-site
INR 1,000,000 - 2,500,000
AWS & Airflow Developer
AWS & Airflow Developer

LPL Financials • Hyderabad

Hybrid
INR 1,200,000 - 1,800,000
Senior Developer, Data Engineer
Senior Developer, Data Engineer

Intercontinental Exchange Holdings, Inc. • Hyderabad

On-site
INR 2,500,000 - 4,500,000
Azure Databricks Data Engineer
Azure Databricks Data Engineer

WTW group • Kolkata District

On-site
INR 1,800,000 - 3,400,000
Data Engineer- Apache Airflow
Data Engineer- Apache Airflow

Astek Group • Ernakulam

On-site
INR 1,200,000 - 1,800,000
Senior Developer, Data Engineer
Senior Developer, Data Engineer

ICE • Hyderabad

On-site
INR 1,800,000 - 3,000,000
Azure Data Engineer
Azure Data Engineer

InfoBeans • Indore District, Pune District

On-site
INR 3,000,000 - 5,400,000
Data Engineer (Azure & Databricks)
Data Engineer (Azure & Databricks)

Lufthansa Technik Services India • Bengaluru

On-site
INR 1,200,000 - 1,500,000
Data Engineer ( Azure & Databricks )
Data Engineer ( Azure & Databricks )

Lufthansa Technik AG • Bengaluru

Hybrid
INR 1,800,000 - 3,200,000
Data Engineer (Azure & Databricks)
Data Engineer (Azure & Databricks)

Lufthansa Technik Services India Pvt Ltd • Bengaluru

On-site
INR 1,000,000 - 1,500,000