L2 Databricks Support Engineer

Nsw Consulting Vadoadara; Hyderabad; Hybrid; INR 1,500,000 - 2,000,000; Full time

Job summary

Nsw Consulting Vadoadara is seeking an experienced L2 Databricks Support Engineer to provide production support for Databricks data platforms and workloads from Hyderabad, India. The role focuses on incident management, debugging, and performance tuning across Spark, PySpark, and Delta Lake environments.

The candidate will collaborate with Dev, Cloud, and Data teams, participate in on-call rotations, and contribute to runbooks and post-release validation.

Qualifications

• Experience providing Level-2 production support for Databricks workloads.
• Strong SQL debugging and performance tuning abilities.
• Experience with Delta Lake and data pipelines.
• Familiarity with ITIL-based incident management.

Responsibilities

• Provide Level-2 production support for Databricks workloads, notebooks, jobs, workflows, and clusters.
• Monitor platform health and proactively resolve incidents, alerts, and service disruptions.
• Troubleshoot Databricks job failures, data issues, SQL errors, and performance bottlenecks.
• Perform SQL analysis, debugging, validation, and query optimization.
• Support Delta Lake, ETL/ELT pipelines, and data integration workflows.
• Monitor scheduled jobs and investigate failed or delayed data pipelines.
• Perform Root Cause Analysis (RCA) and implement preventive measures.
• Manage incidents, problems, and changes following ITIL processes.
• Coordinate with Development, Cloud, Data Engineering, and Business teams during major incidents.
• Support production deployments and validate post-release activities.
• Maintain operational documentation, runbooks, and knowledge articles.
• Participate in 24x7 production support/on-call activities as required.

Skills

Databricks

Apache Spark

PySpark

SQL

Delta Lake

ETL/ELT

Cloud platforms

ITIL

Root cause analysis

Production incident management

Tools

Git

CI/CD

Deployment processes

Monitoring tools

Job description

We are looking for an experienced L2 Databricks Support Engineer to provide production support for Databricks data platforms and workloads. The candidate should have strong experience in Databricks, Apache Spark, PySpark, SQL, Delta Lake, ETL/ELT, cloud platforms, and production incident management.

Key Responsibilities

• Provide Level-2 production support for Databricks workloads, notebooks, jobs, workflows, and clusters.
• Monitor platform health and proactively resolve incidents, alerts, and service disruptions.
• Troubleshoot Databricks job failures, data issues, SQL errors, and performance bottlenecks.
• Perform SQL analysis, debugging, validation, and query optimization.
• Support Delta Lake, ETL/ELT pipelines, and data integration workflows.
• Monitor scheduled jobs and investigate failed or delayed data pipelines.
• Perform Root Cause Analysis (RCA) and implement preventive measures.
• Manage incidents, problems, and changes following ITIL processes.
• Coordinate with Development, Cloud, Data Engineering, and Business teams during major incidents.
• Support production deployments and validate post-release activities.
• Maintain operational documentation, runbooks, and knowledge articles.
• Participate in 24x7 production support/on-call activities as required.

Technical Skills

• Strong experience with Databricks Workspace, Jobs, Workflows, Clusters, and Unity Catalog.
• Good knowledge of Apache Spark and PySpark.
• Strong SQL development, troubleshooting, and performance tuning skills.
• Hands-on experience with Delta Lake architecture and data management.
• Experience with Azure Data Lake Storage (ADLS), AWS S3, or equivalent cloud storage.
• Understanding of ETL/ELT processes and data warehouse concepts.
• Knowledge of Git, CI/CD, and deployment processes.
• Experience with monitoring tools, log analysis, and production troubleshooting.
• Understanding of cloud platforms such as Azure, AWS, or GCP.
• Knowledge of data pipeline monitoring and failure recovery processes.