Lead Site Reliability Engineer

Diverse Lynx, LLC; San Francisco (CA); Remote; USD 110,000 - 130,000; Full time

Job summary

Diverse Lynx, LLC is seeking a Lead Site Reliability Engineer in San Francisco, CA with remote options. You will lead initiatives to improve reliability, scalability, and operational excellence across observability platforms.

Responsibilities include designing and operating Splunk infrastructure, managing Elasticsearch clusters, and building end-to-end tracing pipelines using OpenTelemetry and Grafana Tempo. Proficiency in Terraform, scripting, and cloud tools is required.

Qualifications

• Hands-on experience administering Splunk Enterprise or Splunk Cloud.
• Strong knowledge of Splunk SPL.
• Experience with Elasticsearch/ELK, Prometheus, Grafana, Grafana Tempo, distributed tracing, OpenTelemetry, and Kafka.
• Experience implementing metrics, logs, and traces as part of a modern observability strategy.
• Experience with Terraform and Infrastructure as Code.
• Programming experience in Python, Go, Ruby, or Bash.

Responsibilities

• Design, deploy, and operate enterprise observability platforms.
• Build and maintain Splunk Enterprise/Splunk Cloud infrastructure including Indexers, Search Head Clusters, Heavy Forwarders, and Deployment Servers.
• Deploy and operate large-scale Elasticsearch clusters for log analytics and search.
• Design, deploy, and support distributed tracing platforms using Grafana Tempo and OpenTelemetry.
• Build and maintain end-to-end tracing pipelines, instrumentation standards, and trace retention strategies.
• Scale Prometheus, Grafana, Kafka, Tempo, and OpenTelemetry-based monitoring solutions.
• Develop dashboards, alerts, analytics, and trace visualizations using Splunk SPL, Grafana, Kibana, and Tempo.
• Automate infrastructure using Terraform and configuration management tools.

Skills

Observability

IaC

Scripting

Tools

Splunk Enterprise

Splunk Cloud

Elasticsearch

Elastic Stack

Prometheus

Grafana

Grafana Tempo

OpenTelemetry

Kafka

Terraform

Kubernetes

AWS

Azure

Ansible

CI/CD

Service mesh

Job description

Job Title: Lead Site Reliability Engineer

Location: San Francisco, CA (Remote)

Pay Rate: $55/Hr on W2

Job Summary :

You'll lead initiatives that improve reliability, scalability, and operational excellence.

Key Responsibilities

• Design, deploy, and operate enterprise observability platforms.
• Build and maintain Splunk Enterprise/Splunk Cloud infrastructure including Indexers, Search Head Clusters, Heavy Forwarders, and Deployment Servers.
• Deploy and operate large-scale Elasticsearch clusters for log analytics and search.
• Design, deploy, and support distributed tracing platforms using Grafana Tempo and OpenTelemetry.
• Build and maintain end-to-end tracing pipelines, instrumentation standards, and trace retention strategies.
• Scale Prometheus, Grafana, Kafka, Tempo, and OpenTelemetry-based monitoring solutions.
• Develop dashboards, alerts, analytics, and trace visualizations using Splunk SPL, Grafana, Kibana, and Tempo.
• Automate infrastructure using Terraform and configuration management tools.

Required Qualifications

• Hands-on experience administering Splunk Enterprise or Splunk Cloud.
• Strong knowledge of Splunk SPL.
• Experience with Elasticsearch/ELK, Prometheus, Grafana, Grafana Tempo, distributed tracing, OpenTelemetry, and Kafka.
• Experience implementing metrics, logs, and traces as part of a modern observability strategy.
• Experience with Terraform and Infrastructure as Code.
• Programming experience in Python, Go, Ruby, or Bash.

Preferred Qualifications

• Splunk certification.
• Experience with Kubernetes, AWS/Azure/Eligible to workP, Ansible, Consul, CI/CD pipelines, and service mesh technologies.
• Experience supporting FedRAMP or regulated environments.

Disclaimer: Diverse Lynx LLC is an Equal Opportunity Employer. All applicants and employees are evaluated without discrimination, based solely on their qualifications, ability, competence and performance.