Observability Engineer

Goosehead Insurance Agency LLC; Westlake (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

Goosehead Insurance Agency LLC in Westlake, TX is seeking an Observability Engineer to build and improve health, performance, and reliability telemetry across applications and services. You will implement OpenTelemetry instrumentation, dashboards, and alerts, partnering with Cloud & Infrastructure, NOC, and Production Support to speed production diagnosis.

You will design service-health standards, reduce alert noise, and connect telemetry with incident workflows.

Qualifications

• Minimum 3 years hands-on experience in observability or related roles.
• Experience with OpenTelemetry for metrics, logs, and traces.
• Strong troubleshooting and collaboration across teams.
• Bachelor’s degree in IT, CS, or related field preferred.

Responsibilities

• Design, implement, and improve observability for production apps and APIs.
• Instrument applications with OpenTelemetry and structured logging.
• Create dashboards, alerts, and service-health views for clear visibility.
• Align instrumentation with SLIs/SLOs and incident workflows.

Skills

Observability Engineering

SRE / DevOps

OpenTelemetry

Distributed tracing

Automation scripting

Education

Bachelor's degree in IT / CS

Tools

New Relic

Datadog

Dynatrace

Grafana

Prometheus

Job description

Position Overview The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry. The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool. Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.

Key Responsibilities

• Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry.
• Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.
• Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.
• Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).
• Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.
• Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.
• Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.
• Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.
• Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.
• Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.
• Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.
• Maintain service-based error budgets which our development teams use to dictate feature work.

Qualifications

• Minimum 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.
• Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.
• Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.
• Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.
• Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.
• Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.
• Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.
• Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.
• Strong troubleshooting, analytical, communication, and cross-team collaboration skills.
• Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.

Preferred Qualifications

• Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.
• Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.
• Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.
• Experience establishing SLIs/SLOs or service-health measures for production applications.
• Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.
• Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.
• Familiarity with ITIL Incident Management and Problem Management practices.

Since 2003, Goosehead has been changing the face of the insurance industry and making the search for great coverage a piece of cake. We’re committed to finding the best policies and rates, and that starts with our team. Our people are smart, inclusive, hungry for success, and strive everyday to win. There is noble purpose in all our efforts. The work we do matters. Not only are we fixing a broken industry, we get to win professionally and financially while doing it.

If you think you have what it takes to grow with our company, we would love to meet you.