Observability Engineer/lead/Architect

TENARAI TECHNOLOGIES PRIVATE LIMITED; Bengaluru; Hybrid; INR 3,500,000 - 5,500,000; Full time

Job summary

TENARAI TECHNOLOGIES PRIVATE LIMITED seeks an experienced Observability Engineer to lead migration from Datadog to New Relic, implement OpenTelemetry, and integrate cloud telemetry across AWS and other clouds. You will instrument Meridian MCP, set up dashboards, alerts, and runbooks, and ensure scalable observability across multi-tenant environments.

Responsibilities include agent deployment, log parsing, and synthetic checks, with a focus on reliability, performance insights, and cross-team

Qualifications

• 7–10 years in infrastructure, SRE or DevOps roles.
• 4+ years dedicated to observability/monitoring.
• 2+ years hands-on New Relic delivery and migrations from other tools.
• Production OpenTelemetry collector deployments at scale.
• Experience instrumenting AWS-heavy, containerized workloads.
• Bachelor's degree in CS/Engineering or equivalent experience.

Responsibilities

• Agent deployment and migration from Datadog to New Relic on multiple nodes.
• Instrument Meridian MCP pipeline end to end with observability tooling.
• Configure OpenTelemetry collectors and dual-ship data across systems.
• Add New Relic exporters/receivers alongside Prometheus/OTel flows.
• Configure cloud telemetry across AWS and other clouds.
• Stand up monitoring for MySQL, PostgreSQL, MongoDB Atlas and Neptune.
• Implement log forwarding to New Relic and pattern extraction for NRQL.
• Draft alert policies per tech and route through Jira/Teams.
• Create runbooks and ensure observability is part of DoD.

Skills

SRE basics

Observability

Containers

Scripting Bash

Python automation

Terraform basics

Stakeholder collaboration

Education

Bachelor's degree in CS/Engineering

Equivalent experience

Tools

New Relic

OpenTelemetry

Linux

Docker

AWS

Prometheus

Thanos

Grafana

Loki

Datadog

Azure Monitor

CloudWatch

Terraform

Jira

Airflow

Kubernetes

Job description

Title

Observability Engineer (New Relic & OpenTelemetry) Primary swim lane Configure (lead) supports Enhance and Scope recommendations Experience 710 years total; 4+ years hands-on observability; 2+ years New Relic

Job Description

1. Key Responsibilities

Agent deployment & migration

• Datadog cut-over for Meridian — Replace the Datadog agent with the New Relic infrastructure/APM agent on the ~54 Meridian nodes (Docker on EC2), updating onboarding scripts so new client instances come up instrumented.
• Meridian MCP from zero — Instrument the MCP pipeline end to end (Airflow, Step Functions, Lambda, Neptune, AgentCore Gateway) — currently no observability exists.
• Pattern selection — Apply the agent-first pattern; where an agent cannot be installed (e.g., SaaS such as HashiCorp Vault, Cloudflare, WorkOS, MongoDB Atlas) fall back to OpenTelemetry, cloud integrations, API polling, SNMP or synthetic checks, and document when to use each.

OpenTelemetry & cloud integrations

• OTel collectors — Add New Relic exporters/receivers alongside existing Prometheus and OTel flows so services dual-ship during migration.
• Cloud telemetry — Configure AWS integrations (CloudWatch metric streams, CloudTrail, EC2, ECS/Fargate, S3, Neptune, Lambda, Step Functions) across the correct AWS accounts; handle Azure Monitor and GCP sources for cross-cloud workflows.
• Database coverage — Stand up MySQL, PostgreSQL, MongoDB Atlas and Neptune monitoring for the four Meridian/MCP data stores.

Operational logs

• Log forwarding — Add New Relic as a destination next to Loki and Azure Monitor for operational (not security-pipeline) logs; keep MDR security telemetry clearly out of scope.
• Parsing & patterns — Build log parsing rules and pattern extraction so logs are queryable in NRQL, reusing prior art from Datadog, Loki and Azure Monitor.

Alerting, incident routing & validation

• Alert conditions — Draft alert policies and conditions per technology (e.g., MySQL, Airflow, EC2) with owning engineers, tuning for noise versus coverage.
• Jira-first routing — Route alerts into Jira work items of the right severity so existing on-call schedules, paging and Teams notifications apply; do not duplicate on-call in New Relic.
• Service health — Implement synthetic transactions and health-endpoint/webhook patterns that can drive the status page and customer portal service status.
• End-to-end proof — For each component, prove signal ? New Relic ? dashboard ? alert ? Teams + Jira works, then capture it as a runbook/playbook.

Tracking & enablement

• Maintain the Produce/Consume inventory status for each component (metrics, logs, events, traces, agent; dashboard, alert, Teams, Jira, portal, status, MCP).
• Write the playbooks Cyderes engineers will follow so observability becomes part of the "definition of done.

"Service group Technologies

Meridian — Provision Docker, AWS ECR, AWS SSM, Jenkins CI/CD pipelines; SaaS Customer Manager, Update Manager (per-client provisioning)

Meridian — Ingress Tunnel-free and tunnel-dependent connector VMs (~800 connectors), tunnel proxy server, MySQL data cache on AWS EC2 (per-client containers)

Meridian — Process / Analyze Apache Airflow (scheduler/workers) with PostgreSQL backend, Action Manager; Python ML Merger; LLM VM (Ollama, IBM Granite)

Meridian — Visualize / Store Multi-tenant Web UI and API; Cloudflare (WAF/CDN/DNS/DDoS, R2); WorkOS (SSO/SAML); MongoDB Atlas; AWS S3

Meridian MCP Airflow + Merger (produce) ? S3 Uploader (export) ? AWS Step Functions / Lambda staging (transform) ? Amazon Neptune snapshot, bulk load, validation (load) ? Bedrock AgentCore Gateway, Lambda, Claude, MCP clients (consume)

Vault HashiCorp Vault (secrets storage and rotation; SaaS dependency for every service — agentless monitoring)

COAL (Deploy) AWS ECS, AWS Fargate; frontend, backend and worker services

Platform AWS Control Tower, AFT (Account Factory for Terraform) pipeline, Terraform, GitHub Actions, Argo CD, Backstage

Integrations Jira (incidents, on-call schedules, paging), Microsoft Teams, public status page, customer portal, Cyderes MCP

Clouds / legacy AWS (primary, growing under private pricing), Azure and GCP; legacy Prometheus, Thanos, Grafana, Loki, Datadog, Azure Monitor

3. Required Technical Skills

Area Expectation

• New Relic (must) Infrastructure and APM agents, NRQL, log management and parsing, alerts and applied intelligence, synthetics, workflows/destinations, cloud integrations
• OpenTelemetry (must) Collector configuration, receivers/exporters, semantic conventions, SDK/auto-instrumentation basics, dual-shipping
• Linux & containers (must) Linux administration, Docker and Docker Compose, systemd, syslog, agent packaging and rollout
• AWS (must) EC2, ECS/Fargate, Lambda, Step Functions, S3, Neptune, CloudWatch, CloudTrail, IAM for monitoring roles, multi-account setup
• Legacy tooling (must) Working knowledge of Prometheus, Thanos, Grafana, Loki and Datadog to migrate dashboards/alerts and reuse prior art
• Data stores Monitoring MySQL, PostgreSQL, MongoDB Atlas and graph DBs (Neptune); Airflow metrics
• Integrations Jira alert integration and severity mapping; Microsoft Teams; webhooks
• Scripting Bash and Python for automation; comfortable reading Terraform to hand off configs

4. Domain & Functional Knowledge

• SRE and operations practice — SLIs/SLOs, golden signals (rate, errors, duration, saturation), MTTR, alert fatigue management.
• Multi-tenant SaaS — per-client ingress containers versus shared services; tenant-safe dashboards (one customer never sees another's data).
• Security/MDR context — understanding that at Cyderes "logs" often means customer security telemetry; separate operational logs from the product pipeline.
• Observability migrations — running legacy and new platforms in parallel, validating parity, then decommissioning.
• Stakeholder working style — gathering requirements from app developers and getting sign-off on dashboards and alerts.

5. Experience & Qualifications

• 7–10 years in infrastructure, SRE or DevOps roles, including 4+ years dedicated to observability/monitoring.
• 2+ years hands-on New Relic delivery; at least one migration from Datadog, Prometheus/Grafana or similar to New Relic.
• Production OpenTelemetry collector deployments at scale.
• Experience instrumenting AWS-heavy, containerized workloads; exposure to Azure and GCP.
• Bachelor's degree in Computer Science/Engineering or equivalent experience.