Python Airflow Data Engineer

KPI Partners; Hyderabad, Pune District, Bengaluru; Hybrid; INR 1,800,000 - 2,400,000; Full time

Job summary

KPI Partners is seeking a Python Airflow Data Engineer to design and maintain scalable data pipelines. You will build Airflow DAGs, craft complex SQL, and manage ETL/ELT processes across AWS and Azure environments.

The role requires 6+ years of experience, a hybrid work model across Bangalore/Hyderabad and Pune, and collaboration with data stakeholders to ensure data quality and performance.

Qualifications

• 6+ years of experience in data engineering or equivalent role.
• Strong Python and SQL skills with ETL/ELT development.
• Experience with Airflow DAGs, scheduling and orchestration.
• Cloud experience with AWS and Azure is required.
• Knowledge of CI/CD and data quality practices.

Responsibilities

• Design, develop and maintain scalable data pipelines using Python.
• Build, schedule and manage workflows with Airflow DAGs.
• Develop complex SQL queries and optimize database performance.
• Create ETL/ELT processes for large-scale data integration.
• Leverage AWS and Azure for data storage, processing and integration.
• Ensure data quality, reliability and scalability of pipelines.
• Troubleshoot failures and implement monitoring and error handling.
• Collaborate with data architects and business stakeholders.
• Follow coding standards and CI/CD best practices.

Skills

Python

Airflow

SQL

ETL/ELT

AWS

Azure

CI/CD

Tools

Python

Apache Airflow

SQL

Job description

We at KPI Partners hiring for Python Airflow Data Engineers

Experince 6+ Years

Hybrid Work Model - Bangalore/Hyderabad and Pune

Key Responsibilities

• Design, develop and maintain scalable data pipelines using Python.
• Build, schedule and manage workflows using Apache Airflow, including DAG development and orchestration.
• Develop complex SQL queries, optimize database performance and manage data transformations.
• Build and maintain ETL/ELT processes for large-scale data integration.
• Work with AWS cloud services to develop and manage data engineering solutions.
• Leverage Azure cloud services for data storage, integration and processing.
• Ensure data quality, reliability, performance and scalability across data pipelines.
• Troubleshoot pipeline failures and implement monitoring and error-handling mechanisms.
• Collaborate with cross-functional teams, data architects and business stakeholders.
• Follow coding standards, version control and CI/CD best practices.