Support Lead -Data Production & Platform Operations

Multiverse Solutions; Dadri; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

Multiverse Solutions in India delivery centre is seeking a senior production operations lead for the Data Ops Platform cockpit. You will own the production side of the Databricks pipelines, manage 50+ workflows, lead L1/L2 operations, and act as escalation point on all shifts.

This role requires strong SQL/Python, cloud exposure (Azure or AWS), incident management, and coaching 3–5 engineers. The on-site Day-peak ET shift plus on-call responsibilities form the core of the daily schedule.

Qualifications

• Hands-on experience operating production data platforms (Databricks/Spark) and managing 50+ workflows.
• Proven incident, problem, and change discipline with on-call rota design.
• Strong SQL and Python for diagnostics and automation; reading pipeline code and configs.

Responsibilities

• Lead L1/L2 operations for standard Databricks pipelines, monitor status and throughput.
• Own Gate 1 QC: data freshness, schema conformance and early gating.
• Manage production incidents end-to-end: triage, fix, escalate, conduct root-cause analysis.
• Track system health of Databricks, Kafka, orchestration, databases; flag degradations.
• Develop SOPs, runbooks, and onboarding checklists for new use cases.
• Coach and manage 3–5 junior engineers; perform shift handoffs and reviews.
• Optimize Databricks workloads: tuning, sizing, cost management; automate triage.

Skills

Databricks/Spark

L1/L2 operations leadership

SQL

Python

Azure or AWS

Data security and governance

Stakeholder communication

Education

B.E./B.Tech./MCA or equivalent

Tools

Delta Lake

Databricks

Azure

AWS

Job description

About the role

India delivery centre Day-peak ET shift + on-call Own the production side of the Data Ops Platform cockpit. Your stream answers did the workflow run and deliver? you lead L1/L2 operations over the Workflow Monitor and System Health console for standard Databricks pipelines that load vendor market data, run curve-calibration DAGs and publish to trading and risk systems. You own the operational side of Gate 1 (the config-based technical QC engine running every ~10 minutes across 50+ catalog tables), manage production incidents end-to-end, and lead the junior production engineers. You are the senior on-seat presence on the main trading-day shift and the escalation point on all others.

Key responsibilities

• Lead 246/247 L1/L2 operations for standard Databricks pipelines: live workflow status (queued / running / complete / failed), progress and timing across 50+ workflows, and the EoD batch window (~17:42 ET).
• Operate and evolve Gate 1 technical QC: feed arrival and latency, data freshness, row counts vs rolling averages, schema conformance and range checks including the planned move from warn-only alerts to enforced pipeline gating.
• Own incident & problem management for production failures (each linked to its workflow ID and affected entities): triage, drive verify/fix/escalate per SOP, route vendor-side feed issues to vendor management, run root-cause analysis and post-incident reviews, improve MTTR.
• Track System Health for every component the platform depends on Databricks clusters and jobs, Kafka, orchestration engines, databases and raise degradations before they become breaks.
• Own SOPs, runbooks and the knowledge base; run the standard-pipeline onboarding checklist as new use cases join the platform (the framework generalises beyond Phase 1).
• Build, coach and performance-manage the junior production team (3–5 engineers); run shift handoffs and quality reviews; mature the function toward 2027 scorecards, full intra-day L1 and continuous improvement.
• Optimise Databricks workloads: Spark job tuning, cluster sizing and job-cost management, DAG dependency health; champion automation and AI-assisted triage to cut manual toil.

Required skills & experience

• Strong hands-on experience operating production data platforms — Databricks/Spark strongly preferred (cluster management, job optimisation, Delta Lake, workflow orchestration).
• Proven L1/L2 operations leadership: incident/problem/change discipline (ITIL or equivalent), ticketing and observability tooling, on-call rota design; a record of improving operational metrics.
• Solid SQL and Python for diagnostics and automation; comfort reading pipeline code, configs and execution plans.
• Cloud exposure (Azure or AWS); understanding of data security, governance and auditability in a regulated financial environment; clear, proactive stakeholder communication during incidents.

Preferred

• Asset-management / hedge-fund / capital-markets data operations (market-data feeds, EoD batch cycles, vendor management); Kafka; experience leading teams in a GCC or managed-services setting.

Profile

B.E./B.Tech/MCA or equivalent. Calm, decisive, accountable — able to make a defensible deliver-or-hold call and communicate it to demanding front-office stakeholders