Production and Support Engineer

Technogen; Hyderabad; On-site; INR 2,200,000 - 4,200,000; Full time

Job summary

Technogen Hyderabad is seeking an Airflow & Databricks Scheduling SME with 5+ years of experience. The role focuses on designing, maintaining, and optimizing Airflow workflows and Databricks jobs to ensure reliable data pipelines and SLA compliance in a 24/7 enterprise environment.

The candidate will monitor and troubleshoot production jobs, coordinate with Data Engineering and Cloud teams, and drive automation and platform optimization for critical business processes.

Qualifications

• Experience in designing and maintaining Airflow workflows.
• Hands-on Databricks scheduling and job orchestration.
• Strong scripting in Python and SQL for data pipelines.
• Experience with incident and problem management in production.

Responsibilities

• Own Airflow and Databricks scheduling platforms and pipelines.
• Design, monitor, and optimize enterprise data workflows.
• Troubleshoot production issues and ensure SLA adherence.
• Collaborate with Data Engineering, Cloud, and Operations teams.

Education

Bachelor's degree in Computer Science or related field

Tools

Apache Airflow

Azure Databricks

Python

PySpark

SQL

Azure Cloud

Linux/Unix

Git

Monitoring Tools

Job description

Role & responsibilities

Preferred candidate profile

Job Title: Airflow & Databricks SME

Job Location: Hyderabad

Experiences: 5+ Years

Job Type: Full-time

Shift Timings: Rotational Shifts (24/7)

Work Mode: 5 Days Work from Office

Job Summary:

We are looking for a 5 to 10 years of experienced Airflow & Databricks Scheduling SME to manage, support, optimize, and govern enterprise scheduling platforms and workflow orchestration processes. The candidate will be responsible for designing and maintaining Airflow workflows, scheduling Databricks jobs, monitoring critical business data pipelines, resolving production issues, and ensuring SLA compliance across the enterprise data ecosystem.The role requires strong expertise in Apache Airflow, Azure Databricks, workflow scheduling, incident management, problem management, and production support operations.

Scheduling & Workflow Management

• Design, develop, and manage enterprise scheduling workflows using ApacheAirflow and Databricks
• Create, monitor, and maintain Airflow DAGs for batch and real-time data processing.
• Schedule and orchestrate Databricks notebooks, jobs, and workflows.
• Manage complex job dependencies and event-based scheduling.
• Ensure successful execution of daily, weekly, and monthly business-critical jobs.
• Monitor job completion, SLA adherence, and schedule compliance.
• Validate upstream and downstream job dependencies.
• Configure and manage Databricks Job Workflows.
• Monitor cluster performance and job execution.
• Troubleshoot failed Databricks jobs and Spark execution issues.
• Coordinate with Data Engineering teams for workflow enhancements.
• Optimize Databricks job execution time and resource consumption.
• Configure and manage Databricks Job Workflows.
• Schedule Databricks notebooks and ETL pipelines using Airflow.
• Monitor cluster performance and job execution.
• Troubleshoot failed Databricks jobs and Spark execution issues.
• Coordinate with Data Engineering teams for workflow enhancements.
• Optimize Databricks job execution time and resource consumption.

Key Responsibilities Summary

The Scheduling SME will:

• Own Airflow and Databricks scheduling platforms.
• Ensure successful execution of enterprise data workflows.
• Monitor and support business-critical batch processing.
• Handle incidents, outages, and escalations.
• Lead Major Incident Management activities related to scheduling failures.
• Perform RCA and Problem Management.
• Drive automation and platform optimization.
• Ensure SLA adherence and operational excellence.
• Collaborate with Data Engineering, Cloud, Infrastructure, and Operations teams

Mandatory Skills

• Apache Airflow, Azure Databricks
• Workflow Scheduling & Orchestration
• Python, PySpark, SQL
• Azure Cloud, Linux/Unix, Git
• Monitoring Tools (Splunk, Grafana, Dynatrace)

Education & Certifications:

Bachelors degree Computer Science, or related field