Observability Engineer

Goosehead-Insurance; Westlake (OH); On-site; USD 120,000 - 160,000; Full time

Job summary

Goosehead-Insurance is seeking an Observability Engineer to design and enhance production telemetry across applications and services. You will work with software, DevOps, NOC, incident management, and production support to instrument services, improve telemetry, and establish health standards.

You will implement structured logging, distributed tracing, and custom metrics with OpenTelemetry, and build dashboards and alerts to clearly show availability, latency, and impact.

Qualifications

• 3+ years in Observability, SRE, DevOps, or related roles.
• Experience with OpenTelemetry for metrics, logs, and traces.
• Strong ability to design SLIs/SLOs and alerting standards.
• Experience with cloud environments and CI/CD workflows.

Responsibilities

• Design, implement, and improve observability for production applications and services.
• Instrument applications using OpenTelemetry for metrics, logs, and traces.
• Create dashboards, alerts, service maps, and diagnostic views for health visibility.
• Establish standards for instrumentation, monitoring, SLIs, and SLOs.
• Improve alert quality and reduce noise for effective NOC triage.
• Collaborate with DevOps/Cloud teams to connect app telemetry with infrastructure.

Skills

Observability

SRE

DevOps

Incident management

SLI/SLO design

Cloud knowledge

Automation scripting

Python

PowerShell

Bash

Telemetry

Education

Bachelor's degree in IT/CS/Engineering

Equivalent technical experience

Tools

OpenTelemetry

New Relic

Datadog

Dynatrace

Grafana

Prometheus

ServiceNow

Job description

Position OverviewThe Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.Key Responsibilities· Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry.· Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.· Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.· Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).· Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.· Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.· Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.· Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.· Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.· Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.· Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.· Maintain service-based error budgets which our development teams use to dictate feature work.Qualifications· Minimum 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.· Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.· Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.· Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.· Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.· Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.· Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.· Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.· Strong troubleshooting, analytical, communication, and cross-team collaboration skills.· Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.Preferred Qualifications· Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.· Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.· Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.· Experience establishing SLIs/SLOs or service-health measures for production applications.· Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.· Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.· Familiarity with ITIL Incident Management and Problem Management practices.