Observability Engineer (OTEL)

Softility Inc.; Hyderabad; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Softility Inc. in Hyderabad, India, seeks an experienced Lead to guide an offshore engineering team building a unified observability platform for production and non-prod environments on GCP/Anthos.

The role centers on delivering end-to-end visibility with an open-source OTEL stack and a migration path from legacy tools. You will implement the instrumentation, logging, and metrics stack using Prometheus, Grafana, Tempo, OpenSearch, and Data Prepper, while enforcing GitOps with Terraform and

Qualifications

• 8+ years in observability, APM, and logging with OTEL instrumentation.
• Hands-on with Prometheus, Grafana, Tempo, and OpenSearch/ELK.
• Experience migrating from legacy tools like Splunk or New Relic.

Responsibilities

• Lead deployment of the end-to-end open-source observability stack on GCP/Anthos Kubernetes clusters.
• Design and configure the logging pipeline using Data Prepper, OpenSearch, OpenSearch Dashboards, and ElastAlert.
• Implement observability layer to collect metrics and traces via OTEL instruments, routing to Prometheus and Grafana Tempo.
• Set up synthetic and real user monitoring (K6 Synthetics) and infrastructure monitoring (Node Exporter, kube-state-metrics).
• Ensure IaC & GitOps with Terraform and Jenkins pipelines for automated deployments.
• Develop unified Grafana dashboards (SLO/SLI) with alert deduplication and routing via Prometheus/Alertmanager.
• Migrate existing New Relic dashboards/alerts and Splunk logs to Grafana/OpenSearch with parity validation.
• Mentor offshore engineers and collaborate with onshore leads and enterprise architects.

Skills

Observability

OpenTelemetry

Prometheus

Grafana

Tempo

OpenSearch

ELK Stack

Kubernetes

GCP

Anthos

Terraform

Jenkins

K6

Playwright

Puppeteer

Team Leadership

Education

UG/PG IT background

Tools

Jenkins

Terraform

Job description

Shift Timings: Afternoon Shifts (1:00 PM – 10:00 PM IST)

Role Overview: Lead the offshore engineering team in building and deploying a unified observability platform across production environments (GCP/Anthos clusters) and non-prod. The goal is to provide end-to-end visibility, using a modernized, open-source Open Telemetry (OTEL) stack while working to replace legacy monitoring and correlation tools.

Key Responsibilities

• Platform Architecture & Deployment:Lead the deployment of the end-to-end open-sourceobservabilitystack on GCP/Anthos Kubernetes clusters.
• Logging Layer Implementation:Design and configure the logging pipeline using Data Prepper (for data enrichment), OpenSearch (for indexing and storage), OpenSearch Dashboards, and ElastAlert.
• Metrics & Distributed Tracing:Implement theobservabilitylayer to capture metrics and traces from target API clusters using OTEL Agents and Collectors, routing them to Prometheus and Grafana Tempo.
• UX & Infrastructure Monitoring:Set up Synthetic and Real User Monitoring (RUM) using K6 Synthetics, Black box Exporter, and Grafana Faro, alongside infrastructure monitoring via Node Exporter and Kube-state-metrics.
• Configuration as Code (CaC) & GitOps:Ensure allobservabilityinfrastructure, OTEL collector configurations, and dashboards are version-controlled in Git and deployed automatically using Jenkins pipelines and Terraform.
• Dash boarding & Alerting:Develop unified, SLO/SLI-driven Grafana dashboards covering the four golden signals (Latency, Errors, Traffic, and Saturation). Configure alert deduplication, correlation, and routing using Prometheus and Alert manager.
• Migration & Parity Validation:Run parallel validations to map and migrate existing New Relic dashboards/alerts and Splunk logs to the new Grafana/OpenSearch stack, ensuring full parity before legacy platform turn-down.
• Team Leadership:Mentor offshore engineers, identify and bridge skill gaps within the team, and collaborate closely with onshore leads and enterprise architects.

Required Technical & Professional Skills

• CoreObservability: 8+ years of experience inobservability, APM, and logging. Deep expertise in OpenTelemetry (OTEL) instrumentation (Agents/Collectors) is mandatory.
• Open-Source Stack:Strong hands-on experience with Prometheus, Grafana, Tempo, and OpenSearch/ELK stack.
• Legacy Tool Migration:Proven experience working with or migrating away from commercial tools like Splunk and New Relic.
• Cloud & Containers:Strong proficiency in Kubernetes orchestration (GCP/Anthos environment experience preferred) and micro services architecture.
• Infrastructure as Code (IaC):Advanced skills in Terraform and CI/CD pipelines (Jenkins) for automated deployments.
• Scripting & Synthetics:Experience with synthetic transaction scripting (K6, Playwright, or Puppeteer).
• Leadership:Demonstrated experience leading offshore technical teams, managing project milestones, and driving architectural best practices. Excellent communication skills for onshore-offshore coordination.

Education : Preferred any IT Background –UG OR PG