Airflow And Databricks SME

Solugenix

Hyderabad

Presencial

INR 2 000 000 - 3 200 000

Tempo integral

14 dias+

Recebe uma resposta deste empregador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

Solugenix in Hyderabad is seeking an Airflow and Databricks Scheduling SME to design, manage, and optimize enterprise scheduling workflows. You will oversee Airflow DAGs, schedule Databricks jobs, monitor critical pipelines, and drive incident resolution with a focus on SLA compliance.

The role requires strong expertise in Apache Airflow, Azure Databricks, and production support operations, with collaborative engagement across Data Engineering and Cloud teams.

Qualificações

5+ years of experience with Airflow & Databricks scheduling and production support.
Strong expertise in Apache Airflow, Azure Databricks, and workflow orchestration.
Experience in incident management, RCA, problem management, and SLA monitoring.

Responsabilidades

Own Airflow and Databricks scheduling platforms and ensure reliable data workflows.
Monitor and support enterprise batch processing and scheduled jobs.
Handle incidents, outages, and service escalations with root cause analysis.
Drive automation and platform optimization to improve SLA adherence.
Collaborate with Data Engineering, Cloud, and Operations teams for improvements.
Optimize job execution time and resource usage across pipelines.

Conhecimentos

Airflow

Azure Databricks

Workflow Scheduling

Python

PySpark

SQL

Azure Cloud

Linux/Unix

Git

Monitoring Tools

Formação académica

Bachelor's degree in Computer Science

Descrição da oferta de emprego

Job Title: Airflow And Databricks SME

Job Location: Hyderabad

Experience: 5+ Years

Job Type: Full-time

Shift Timings: Rotational Shifts (24/7)

Work Mode: 5 Days Work from Office

Job Summary:

We are looking for a 5 to 8 years of experience Airflow & Databricks Scheduling SME to manage, support, optimize, and govern enterprise scheduling platforms and workflow orchestration processes. The candidate will be responsible for designing and maintaining Airflow workflows, scheduling Databricks jobs, monitoring critical business data pipelines, resolving production issues, and ensuring SLA compliance across the enterprise data ecosystem. The role requires strong expertise in Apache Airflow, Azure Databricks, workflow scheduling, incident management, problem management, and production support operations.

Scheduling & Workflow Management

Design, develop, and manage enterprise scheduling workflows using Apache Airflow and Databricks
Create, monitor, and maintain Airflow DAGs for batch and real-time data processing.
Schedule and orchestrate Databricks notebooks, jobs, and workflows.
Manage complex job dependencies and event-based scheduling.
Ensure successful execution of daily, weekly, and monthly business-critical jobs.
Monitor job completion, SLA adherence, and schedule compliance.
Validate upstream and downstream job dependencies.
Configure and manage Databricks Job Workflows.
Monitor cluster performance and job execution.
Troubleshoot failed Databricks jobs and Spark execution issues.
Coordinate with Data Engineering teams for workflow enhancements.
Optimize Databricks job execution time and resource consumption.
Configure and manage Databricks Job Workflows.
Schedule Databricks notebooks and ETL pipelines using Airflow.
Monitor cluster performance and job execution.
Troubleshoot failed Databricks jobs and Spark execution issues.
Coordinate with Data Engineering teams for workflow enhancements.
Optimize Databricks job execution time and resource consumption.