Airflow & Databricks SME

Solugenix; Hyderabad; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Solugenix is seeking an Airflow & Databricks Scheduling SME with 5+ years of experience to manage and optimize enterprise scheduling platforms. You will design Airflow workflows, schedule Databricks jobs, monitor critical data pipelines, and ensure SLA compliance across the data ecosystem.

Responsibilities include incident management, RCA, and collaboration with Data Engineering teams to improve workflows and reduce execution time. This is a full-time, on-site role based in Hyderabad, India.

Qualifications

• 5+ years of experience in Airflow & Databricks scheduling and production support.
• Strong skills in designing, deploying, and monitoring enterprise data workflows.
• Experience with incident management, RCA, and problem management.

Responsibilities

• Own Airflow and Databricks scheduling platforms and workflows.
• Design, monitor, and optimize batch and real-time data pipelines.
• Manage incidents, outages, and escalations with RCA.
• Coordinate with Data Engineering and Cloud teams for improvements.
• Ensure SLA adherence and operational excellence.
• Drive automation and platform optimization.

Skills

Workflow Scheduling & Orchestration

Python, PySpark, SQL

Monitoring Tools (Splunk, Grafana, Dyn

Education

Bachelor's degree Computer Science, or related field

Tools

Apache Airflow

Azure Databricks

Databricks

Job description

Solugenix is a leader in IT services, delivering cutting-edge technology solutions, exceptional talent, and managed services to global enterprises. With extensive expertise in highly regulated and complex industries, we are a trusted partner for integrating advanced technologies with streamlined processes. Our solutions drive growth, foster innovation, and ensure compliance- providing clients with reliability and a strong competitive edge.

Recognized as a 2024 Top Workplace, Solugenix is proud of its inclusive culture and unwavering commitment to excellence. Our recent expansion, with new offices in the Dominican Republic, Jakarta, and the Philippines, underscores our growing global presence and ability to offer world-class technology solutions. Partnering with Solugenix means more than just business- it means having a dedicated ally focused on your success in todays fast-evolving digital world.

Job Title: Airflow & Databricks SME

Experiences: 5+ Years

Job Type: Full-time

Shift Timings: Rotational Shifts (24/7)

Work Mode: 5 Days Work from Office

Job Summary:

We are looking for a 5 to 8 years of experienced Airflow & Databricks Scheduling SME to manage, support, optimize, and govern enterprise scheduling platforms and workflow orchestration processes. The candidate will be responsible for designing and maintaining Airflow workflows, scheduling Databricks jobs, monitoring critical business data pipelines, resolving production issues, and ensuring SLA compliance across the enterprise data ecosystem. The role requires strong expertise in Apache Airflow, Azure Databricks, workflow scheduling, incident management, problem management, and production support operations.

Scheduling & Workflow Management

• Design, develop, and manage enterprise scheduling workflows using ApacheAirflow and Databricks
• Create, monitor, and maintain Airflow DAGs for batch and real-time data processing.
• Schedule and orchestrate Databricks notebooks, jobs, and workflows.
• Manage complex job dependencies and event-based scheduling.
• Ensure successful execution of daily, weekly, and monthly business-critical jobs.
• Monitor job completion, SLA adherence, and schedule compliance.
• Validate upstream and downstream job dependencies.
• Configure and manage Databricks Job Workflows.
• Monitor cluster performance and job execution.
• Troubleshoot failed Databricks jobs and Spark execution issues.
• Coordinate with Data Engineering teams for workflow enhancements.
• Optimize Databricks job execution time and resource consumption.
• Configure and manage Databricks Job Workflows.
• Schedule Databricks notebooks and ETL pipelines using Airflow.
• Monitor cluster performance and job execution.
• Troubleshoot failed Databricks jobs and Spark execution issues.
• Coordinate with Data Engineering teams for workflow enhancements.
• Optimize Databricks job execution time and resource consumption.

Key Responsibilities Summary

The Scheduling SME will:

• Own Airflow and Databricks scheduling platforms.
• Ensure successful execution of enterprise data workflows.
• Monitor and support business-critical batch processing.
• Handle incidents, outages, and escalations.
• Lead Major Incident Management activities related to scheduling failures.
• Perform RCA and Problem Management.
• Drive automation and platform optimization.
• Ensure SLA adherence and operational excellence.
• Collaborate with Data Engineering, Cloud, Infrastructure, and Operations teams

Mandatory Skills

• Workflow Scheduling & Orchestration
• Python, PySpark, SQL
• Monitoring Tools (Splunk, Grafana, Dynatrace)

Education & Certifications:

Bachelor's degree Computer Science, or related field