Observability (O11y) Engineer (Enterprise Observability Platform)

Carnation Infotech Pvt Ltd.; Hyderabad, Pune District, Bengaluru; Hybrid; INR 2,800,000 - 4,200,000; Full time

Job summary

Carnation Infotech Pvt Ltd. in Hyderabad is seeking an Observability Architect to lead design and implementation of Splunk Observability Cloud solutions. You will shape OTEL pipelines, instrumentation strategies, and service health across microservices.

You will collaborate with SRE/DevOps, define SLIs/SLOs, build dashboards and alerts, and guide teams through onboarding, best practices, and cloud-native observability initiatives.

Qualifications

• Hands-on experience implementing Splunk Observability Cloud (APM, Infra Monitoring, RUM, Log Observer).
• Strong understanding of OpenTelemetry (collectors, processors, exporters, instrumentation).
• Experience with distributed tracing, span analysis, and service dependency mapping.
• Familiarity with Splunk Enterprise and ITSI for integrated observability use cases.
• Strong Linux/Unix skills and scripting (Python, Bash).
• Experience with Kubernetes, Docker, and cloud platforms (AWS, and Azure).

Responsibilities

• Design and implement observability solutions using Splunk Observability Cloud (APM, Infra Monitoring, RUM, Log Observer).
• Build and optimize OpenTelemetry (OTel) collector pipelines for metrics, traces, and logs.
• Define instrumentation strategies for applications (auto-instrumentation, SDKs, agents).
• Develop service maps, dependency graphs, and distributed tracing configurations.
• Partner with application teams to onboard services into O11y and ITSI.
• Define SLIs/SLOs, KPIs, and service health indicators.
• Build dashboards, detectors, alerts, and anomaly detection rules tailored to each service.
• Ensure consistent tagging, metadata, and observability standards across teams.
• Implement ITSI service models, KPIs, correlation searches, and glass tables.
• Integrate O11y telemetry with Splunk Enterprise and ITSI for unified service health.
• Collaborate with Splunk Core SMEs to align ingestion, retention, and data modeling.
• Deploy and configure OTel collectors across VMs, containers, Kubernetes, and cloud services.
• Implement processors, exporters, and routing logic for efficient telemetry flow.
• Optimize pipelines for performance, cost, and reliability.
• Develop onboarding guides, instrumentation patterns, and observability standards.
• Conduct workshops, demos, and training sessions for engineering teams.
• Provide architectural guidance on observability adoption across the enterprise.
• Work with SRE, DevOps, Cloud, and Application teams to embed observability into CI/CD and deployment workflows.
• Support modernization initiatives such as microservices, Kubernetes, and cloud migrations.
• Influence platform roadmap and observability strategy.
• Splunk Observability or OTel certifications.
• Experience with Prometheus, Grafana, Datadog, New Relic, or similar tools.
• Familiarity with Terraform, Ansible, GitOps, or CI/CD pipelines.
• Experience in large enterprise environments.

Skills

Splunk Observability Cloud

OpenTelemetry

Distributed tracing

Linux/Unix

Python

Bash

Kubernetes

Docker

AWS

Azure

Tools

Splunk Observability Cloud

OTel

Splunk Enterprise

ITSI

Kubernetes

Docker

Terraform

Ansible

GitOps

Prometheus

Grafana

New Relic

Job description

Key Responsibilities:

Observability Architecture & Implementation

• Design and implement observability solutions using Splunk Observability Cloud (APM, Infrastructure Monitoring, RUM, Log Observer).
• Build and optimize OpenTelemetry (OTel) collector pipelines for metrics, traces, and logs.
• Define instrumentation strategies for applications (auto-instrumentation, SDKs, agents).
• Develop service maps, dependency graphs, and distributed tracing configurations.

Application & Service Onboarding

• Partner with application teams to onboard services into O11y and ITSI.
• Define SLIs/SLOs, KPIs, and service health indicators.
• Build dashboards, detectors, alerts, and anomaly detection rules tailored to each service.
• Ensure consistent tagging, metadata, and observability standards across teams.

ITSI & Splunk Core Integration

• Implement ITSI service models, KPIs, correlation searches, and glass tables.
• Integrate O11y telemetry with Splunk Enterprise and ITSI for unified service health.
• Collaborate with Splunk Core SMEs to align ingestion, retention, and data modeling.

Telemetry Pipeline Engineering

• Deploy and configure OTel collectors across VMs, containers, Kubernetes, and cloud services.
• Implement processors, exporters, and routing logic for efficient telemetry flow.
• Optimize pipelines for performance, cost, and reliability.

Enablement & Best Practices

• Develop onboarding guides, instrumentation patterns, and observability standards.
• Conduct workshops, demos, and training sessions for engineering teams.
• Provide architectural guidance on observability adoption across the enterprise.

Cross-Functional Collaboration

• Work with SRE, DevOps, Cloud, and Application teams to embed observability into CI/CD and deployment workflows.
• Support modernization initiatives such as microservices, Kubernetes, and cloud migrations.
• Influence platform roadmap and observability strategy.

Required Skills & Experience:

Technical Skills

• Hands-on experience implementing Splunk Observability Cloud (APM, Infra Monitoring, RUM, Log Observer).
• Strong understanding of OpenTelemetry (collectors, processors, exporters, instrumentation).
• Experience with distributed tracing, span analysis, and service dependency mapping.
• Familiarity with Splunk Enterprise and ITSI for integrated observability use cases.
• Strong Linux/Unix skills and scripting (Python, Bash).
• Experience with Kubernetes, Docker, and cloud platforms (AWS, and Azure).

Implementation & Architecture Skills

• Experience designing observability solutions for microservices, APIs, and distributed systems.
• Ability to guide teams through instrumentation and telemetry onboarding.
• Strong understanding of SLIs/SLOs, service health scoring, and reliability engineering.

Soft Skills

• Excellent communication and stakeholder-facing skills.
• Ability to lead workshops and influence engineering teams.
• Strong documentation and architectural thinking.

Preferred Qualifications

• Splunk Observability or OTel certifications.
• Experience with Prometheus, Grafana, Datadog, New Relic, or similar tools.
• Familiarity with Terraform, Ansible, GitOps, or CI/CD pipelines.
• Experience in large enterprise, utilities, financial, or public sector environments.