Apache Airflow Architect

Airo Digital Labs, LLC; Mumbai; On-site; INR 3,000,000 - 5,400,000; Full time

Job summary

Airo Digital Labs, LLC in Mumbai seeks an experienced Airflow/Python engineer to design and maintain data pipelines. You will implement DAGs, ETL jobs, and custom operators, collaborating with teams to translate business needs into robust workflows.

The role requires strong Python, SQL, and data warehousing knowledge, plus experience with Git. You'll contribute to a GitHub-driven environment and ensure tests and reliability across pipelines.

Qualifications

• Proficiency in Python for Airflow development.
• Knowledge of Airflow DAGs, operators, and scheduling.
• Experience with SQL and database design for data pipelines.
• Understanding ETL principles and best practices.
• Familiarity with data warehousing and data lake technologies.
• Version control experience with Git or similar.

Responsibilities

• Design and develop DAGs to orchestrate workflows (data loading, transformation, reporting).
• Implement ETL jobs for extraction, transformation, loading.
• Manage and configure workflows for efficiency and reliability.
• Write custom operators, sensors, hooks for Airflow integration.
• Collaborate with teams to translate business needs into Airflow workflows.
• Contribute to a GitHub-driven environment and follow coding standards.
• Write unit and end-to-end tests for Airflow pipelines.
• Work with data sources and storage like databases, cloud services, data lakes.

Skills

Python

Airflow architecture

SQL

ETL processes

Data warehousing

Data lakes

Git

Job description

Join Airo Digital Labs and build your career with a leader in emerging digital technologies such as robotic process automation, conversational AI, machine learning, the internet of things, voice-based technologies, and cloud enablement. At Airo, we offer you competitive benefits and compensation packages along with the opportunity to learn on the job, develop knowledge of the process, and grow your career. What’s more, you will have fun as you solve some of the most complex business problems.

Responsibilities:

• Design and develop DAGs: Create and maintain DAGs to orchestrate workflows, including data loading, transformation, and reporting.
• Implement ETL jobs: Develop and deploy ETL jobs for data extraction, transformation, and loading.
• Manage and configure workflows: Create, manage, and configure data pipelines, ensuring their efficiency and reliability.
• Write custom operators, sensors, and hooks: Develop custom components for Airflow to extend its functionality and integrate with specific tools and systems.
• Collaborate with teams: Work with other data engineers and stakeholders to understand business needs and translate them into Airflow workflows.
• Contribute to a GitHub-driven environment: Follow coding standards and contribute to a collaborative development process.
• Write unit and end-to-end tests: Ensure the reliability and quality of Airflow pipelines.
• Work with data sources and storage: Integrate Airflow with various databases, cloud services, and data lakes.

Years of Experience

9+ Years.

Skills Required:

• Proficiency in Python: Strong programming skills in Python, the language used for Airflow.
• Understanding of Airflow architecture and concepts: Knowledge of Airflow’s components, DAGs, operators, and scheduling mechanisms.
• Experience with SQL and database design: Familiarity with SQL for data manipulation and database design for managing data pipelines.
• Experience with ETL processes: Understanding of ETL principles and best practices.
• Knowledge of data warehousing and data lakes: Familiarity with data warehousing concepts and data lake technologies.
• Familiarity with version control systems: Experience with Git and other version control systems for code management.