Observability Engineer

Goosehead Insurance Agency; Westlake (TX); On-site; USD 120,000 - 170,000; Full time

Job summary

Goosehead Insurance is seeking an Observability Engineer to build and enhance the health, performance, and reliability of applications and services. You will instrument telemetry, improve alerts, and work with Cloud & Infrastructure to align observability with deployments and cloud platforms.

The role emphasizes reducing alert noise, building dashboards, and supporting incident response across Engineering, NOC, and Production Support. Strong scripting and OpenTelemetry expertise are required.

Qualifications

• Minimum 3 years of hands-on experience in Observability Engineering or related roles.
• Experience with OpenTelemetry for metrics, logs, and traces.
• Strong understanding of SLIs/SLOs and alerting design.
• Knowledge of cloud environments and telemetry integration.

Responsibilities

• Design, implement, and improve observability for production applications, APIs, and services using OpenTelemetry.
• Instrument applications with OpenTelemetry for structured logging, tracing, and metrics.
• Create and maintain dashboards, alerts, and diagnostic views to reveal availability and latency.
• Establish standards for instrumentation, monitoring coverage, and service health metrics.
• Collaborate with NOC, Incident Management, and Production Support to speed diagnosis and reduce alert noise.
• Integrate observability data with ITSM workflows and CI/CD pipelines.

Skills

Observability engineering

SRE / DevOps

OpenTelemetry

Distributed tracing

Application performance monitoring

Scripting (PowerShell, Python, Bash)

Education

Bachelor's degree in IT/CS/Engineering

Tools

New Relic

Grafana

Datadog

Prometheus

ServiceNow

Job description

Position Overview

The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.

The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.

Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.

Key Responsibilities

• Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry.
• Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.
• Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.
• Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).
• Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.
• Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.
• Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.
• Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.
• Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.
• Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.
• Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.
• Maintain service-based error budgets which our development teams use to dictate feature work.

Qualifications

• Minimum 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.
• Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.
• Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.
• Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.
• Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.
• Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.
• Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.
• Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.
• Strong troubleshooting, analytical, communication, and cross-team collaboration skills.
• Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.

Preferred Qualifications

• Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.
• Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.
• Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.
• Experience establishing SLIs/SLOs or service-health measures for production applications.
• Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.
• Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.
• Familiarity with ITIL Incident Management and Problem Management practices.