Observability Engineer

Goosehead Insurance; Texas; On-site; USD 90,000 - 120,000; Full time

Job summary

Goosehead Insurance seeks an Observability Engineer to enhance the health, performance, and reliability of Goosehead applications and services. You will instrument services, improve telemetry, and collaborate with Cloud & Infrastructure to connect telemetry across platforms.

The role emphasizes building dashboards, alerts, SLIs/SLOs, and rapid production diagnosis, with OpenTelemetry and New Relic as core tools.

Qualifications

• 3+ years in Observability/SRE/DevOps or related roles, 4-5 years preferred.
• Hands-on experience with APM, metrics, logging, tracing, dashboards, and production troubleshooting.
• Experience instrumenting apps with OpenTelemetry for metrics, logs, and traces.
• Understanding of APIs, distributed apps, dependencies, and performance issues.

Responsibilities

• Design, implement, and improve observability for production applications and services using OpenTelemetry for metrics, logs, and traces.
• Partner with Engineering to instrument apps with structured logging and distributed tracing.
• Create service-health dashboards, alerts, maps, and diagnostic views to clarify availability and impact.
• Establish instrumentation, monitoring coverage, alerting, and SLI/SLO standards.
• Improve alert quality and reduce noise to support NOC triage and escalation.
• Support Incident Management during significant events by correlating telemetry and deployments.
• Collaborate with Problem Management to close observability gaps after incidents.
• Connect app observability with CI/CD, cloud, and platform telemetry with Cloud & Infrastructure.
• Build scripts and automation (PowerShell, Python, Bash, APIs) to enhance telemetry and workflows.
• Contribute to ITSM integration plans, including future ServiceNow capabilities.
• Maintain practical standards and reusable patterns for Engineering and Technology Operations.

Skills

OpenTelemetry

Observability

SRE

DevOps

Troubleshooting

Cross-team collaboration

Education

Bachelor's degree in IT or related

Tools

PowerShell

Python

Bash

New Relic

Grafana

Prometheus

Job description

The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands‑on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.

The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.

Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.

Key Responsibilitie s

• Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry.
• Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.
• Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.
• Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).
• Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.
• Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.
• Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.
• Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.
• Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.
• Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.
• Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.
• Maintain service-based error budgets which our development teams use to dictate feature work.

Qualifications

• Minimum 3 years of hands‑on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.
• Hands‑on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.
• Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.
• Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.
• Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.
• Hands‑on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.
• Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service‑health concepts.
• Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.
• Strong troubleshooting, analytical, communication, and cross‑team collaboration skills.
• Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.

Preferred Qualifications

• Hands‑on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.
• Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.
• Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.
• Experience establishing SLIs/SLOs or service‑health measures for production applications.
• Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.
• Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.
• Familiarity with ITIL Incident Management and Problem Management practices.