SRE Lead (NOC Operations)

Phykon Solutions; Thiruvananthapuram; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Phykon Solutions seeks an experienced SRE/NOC leader to architect and own the fleet observability and data-center operations across distributed edge sites. You will drive incident command, runbooks, and ongoing improvements, coordinating with data centers, telecoms, and vendors to maintain service continuity.

The role requires deep expertise in Kubernetes, Linux, Prometheus, and Grafana, plus strong on-call leadership and stakeholder communication across engineering and business units.

Qualifications

• 8+ years in SRE, production engineering, or senior NOC roles with strong platform depth.
• Production Kubernetes and Linux experience, ideally in edge or distributed environments.
• Ownership-level experience with Prometheus and Grafana — designing scrape architectures and alert rules, not just reading dashboards.
• Hands-on experience with data center operations — running production infrastructure across data center, colocation, or distributed edge sites.
• Strong practical networking: TCP/IP, NAT/CGNAT, tunnels, and degraded-link diagnosis.
• Demonstrated on-call leadership and Sev1 incident command in production environments.
• Proven ability to codify standards, runbooks, and certification programs for operations teams.
• Strong analytical, troubleshooting, and documentation skills, with excellent communication across engineering and business stakeholders.
• Strong ownership mindset with the ability to independently drive incidents, projects, and operational improvements to completion.
• Ability to remain calm, structured, and decisive during high-severity incidents and customer escalations.
• Excellent stakeholder management and communication skills, including the ability to communicate complex technical concepts to both technical and non-technical audiences.
• Experience interacting directly with enterprise customers, vendors, and senior stakeholders in operational environments.

Responsibilities

• Architect the monitoring stack the NOC runs on scrape architecture, alert rules, dashboards, and SLOs rather than only consuming it.
• Own end-to-end visibility of the connected fleet across data center, edge Kubernetes, Linux, and network layers.
• Continuously tune alerting to reduce noise and catch degradation early.
• Oversee the health and availability of data center and colocation infrastructure — compute, storage, network, power, and cooling dependencies.
• Coordinate with data center operators, colocation providers, and remote-hands teams for physical interventions, maintenance windows, and capacity changes.
• Manage hardware fault handling, RMA workflows, and site-level incident response across distributed data center and edge sites.
• Coordinate with telecommunications providers, colocation partners, cloud vendors, and infrastructure service providers to resolve service-impacting issues and maintain operational continuity.
• Serve as incident commander on Sev1/P1 incidents, and drive them to resolution within SLA.
• Perform deep diagnosis and root-cause analysis on the running system using logs, metrics, and telemetry.
• Diagnose complex issues across: Network connectivity, routing, NAT/CGNAT, and tunnels
• Degraded or intermittent links to field-deployed hardware
• Production Kubernetes and Linux platform dependencies at the edge
• Assume end-to-end ownership of major incidents, service disruptions, and customer escalations until resolution and formal closure.
• Act as the highest operational escalation point within the NOC for complex technical and service‑impacting incidents.
• Provide timely updates to internal stakeholders, leadership teams, and customers during major incidents and service outages.
• Codify diagnostic procedures, escalation paths, and operational standards that make the NOC repeatable and scalable.
• Own the operator‑certification layer that qualifies operators to run and support the fleet.
• Develop and continuously improve runbooks and operational procedures inherited by L1/L2 teams.
• Maintain escalation hygiene: reserve L4 product engineering for genuinely complex issues requiring code or firmware changes.
• Clearly define problem scope, business impact, and affected systems when escalating, with supporting logs and evidence.
• Facilitate technical communication across engineering and operations teams during major incidents, providing timely stakeholder updates.
• Lead post‑incident reviews and drive corrective actions to closure.
• Identify recurring incidents and contribute preventive improvements and monitoring optimization.
• Act as primary technical point of contact for customers, partners, vendors, and internal stakeholders on operational and service‑related matters.
• Participate in customer meetings, operational reviews, service reviews, and technical discussions as a subject matter expert.
• Provide clear, professional, and executive‑level communication during major incidents, service disruptions, and operational escalations.
• Build and maintain strong working relationships with customer technical teams while driving accountability and service excellence.
• Provide technical leadership, mentoring, and guidance to L1 and L2 engineers.
• Drive knowledge sharing, operational maturity, and continuous improvement initiatives across the NOC function.
• Support the development and maintenance of operator certification programs and competency frameworks.
• Participate in recruitment, onboarding, training, and capability development activities as the NOC organization scales.

Skills

SRE experience

Kubernetes

Linux

Prometheus & Grafana

On-call leadership

Incident command

Networking basics

Runbooks & standards

Documentation

Stakeholder communication

Tools

Prometheus

Grafana

Terraform

Job description

Key Responsibilities

1. Fleet Health & Observability

• Architect the monitoring stack the NOC runs on scrape architecture, alert rules, dashboards, and SLOs rather than only consuming it.
• Own end‑to‑end visibility of the connected fleet across data center, edge Kubernetes, Linux, and network layers.
• Continuously tune alerting to reduce noise and catch degradation early.

2. Data Center & Infrastructure Operations

• Oversee the health and availability of data center and colocation infrastructure compute, storage, network, power, and cooling dependencies — supporting the fleet.
• Coordinate with data center operators, colocation providers, and remote-hands teams for physical interventions, maintenance windows, and capacity changes.
• Manage hardware fault handling, RMA workflows, and site‑level incident response across distributed data center and edge sites.
• Coordinate with telecommunications providers, colocation partners, cloud vendors, and infrastructure service providers to resolve service‑impacting issues and maintain operational continuity.

3. Incident Management & Response

• Serve as incident commander on the hardest escalations, including Sev1/P1 incidents, and drive them to resolution within SLA.
• Perform deep diagnosis and root‑cause analysis on the running system using logs, metrics, and telemetry.
• Diagnose complex issues across:
• Network connectivity, routing, NAT/CGNAT, and tunnels
• Degraded or intermittent links to field‑deployed hardware
• Production Kubernetes and Linux platform dependencies at the edge
• Assume end‑to‑end ownership of major incidents, service disruptions, and customer escalations until resolution and formal closure.
• Act as the highest operational escalation point within the NOC for complex technical and service‑impacting incidents.
• Provide timely updates to internal stakeholders, leadership teams, and customers during major incidents and service outages.

4. Standards, Runbooks & Operator Certification

• Codify diagnostic procedures, escalation paths, and operational standards that make the NOC repeatable and scalable.
• Own the operator‑certification layer that qualifies operators to run and support the fleet.
• Develop and continuously improve runbooks and operational procedures inherited by L1/L2 teams.

5. Escalation & Coordination

• Maintain escalation hygiene: reserve L4 product engineering for genuinely complex issues requiring code or firmware changes.
• Clearly define problem scope, business impact, and affected systems when escalating, with supporting logs and evidence.
• Facilitate technical communication across engineering and operations teams during major incidents, providing timely stakeholder updates.

6. Continuous Service Improvement

• Lead post‑incident reviews and drive corrective actions to closure.
• Identify recurring incidents and contribute preventive improvements and monitoring optimization.

7. Client & Stakeholder Engagement

• Act as a primary technical point of contact for customers, partners, vendors, and internal stakeholders on operational and service‑related matters.
• Participate in customer meetings, operational reviews, service reviews, and technical discussions as a subject matter expert.
• Provide clear, professional, and executive‑level communication during major incidents, service disruptions, and operational escalations.
• Build and maintain strong working relationships with customer technical teams while driving accountability and service excellence.

8. Leadership & Team Development

• Provide technical leadership, mentoring, and guidance to L1 and L2 engineers.
• Drive knowledge sharing, operational maturity, and continuous improvement initiatives across the NOC function.
• Support the development and maintenance of operator certification programs and competency frameworks.
• Participate in recruitment, onboarding, training, and capability development activities as the NOC organization scales.

Required Skills & Qualifications

• 8+ years in SRE, production engineering, or senior NOC roles with strong platform depth (not generalist IT‑NOC).
• Production Kubernetes and Linux experience, ideally in edge or distributed environments rather than pure cloud.
• Ownership‑level experience with Prometheus and Grafana — designing scrape architectures and alert rules, not just reading dashboards.
• Hands‑on experience with data center operations — running production infrastructure across data center, colocation, or distributed edge sites.
• Strong practical networking: TCP/IP, NAT/CGNAT, tunnels, and degraded‑link diagnosis.
• Demonstrated on‑call leadership and Sev1 incident command in production environments.
• Proven ability to codify standards, runbooks, and certification programs for operations teams.
• Strong analytical, troubleshooting, and documentation skills, with excellent communication across engineering and business stakeholders.
• Strong ownership mindset with the ability to independently drive incidents, projects, and operational improvements to completion.
• Ability to remain calm, structured, and decisive during high‑severity incidents and customer escalations.
• Excellent stakeholder management and communication skills, including the ability to communicate complex technical concepts to both technical and non‑technical audiences.
• Experience interacting directly with enterprise customers, vendors, and senior stakeholders in operational environments.

Category

Tools

Observability & Telemetry

Prometheus, Grafana, and related monitoring/alerting platforms

Platform

Kubernetes, Linux (edge/distributed deployments)

Automation

Infrastructure-as-Code tooling (advantage)

Incident & Collaboration

Incident management, ticketing, and on‑call platforms

Preferred Qualifications

• Greenfield NOC build experience — standing up a NOC, runbooks, and on‑call from scratch (strongest positive signal).
• Prior NOC/SRE experience supporting a hardware or field‑deployed product.
• Exposure to OT/BMS environments and comfort bridging IT and industrial systems.
• Relevant certifications (Kubernetes, Linux, networking, or cloud) are an advantage.
• Highly desired: Proven experience building or scaling a Network Operations Center (NOC) from the ground up, including monitoring strategy, runbooks, escalation models, operational processes, and team development.

Work Environment

• Operates within a 247 Network Operations Center supporting a live fleet of field‑deployed systems and distributed data center infrastructure.
• May require participation in rotational shifts, weekend coverage, planned maintenance activities, and on‑call support arrangements.
• Serves as a key escalation point for critical incidents and major customer‑impacting events.
• Acts as a primary technical point of contact for customers, partners, and internal stakeholders during operational reviews and service incidents.
• Expected to perform effectively in high‑pressure operational environments while maintaining clear communication, leadership, and decision‑making capabilities.
• Provides technical leadership, mentoring, and operational guidance to L1 and L2 support teams.