Lead Site Reliability Engineer – Observability

Rishi Writes; United States; Remote; USD 140,000 - 180,000; Full time

Job summary

Rishi Writes is seeking a Lead Site Reliability Engineer – Observability to design, build, and run enterprise observability platforms across a large-scale cloud environment. You will lead initiatives to improve reliability, scalability and operational excellence while working remotely from the United States.

You will own Splunk infrastructure, Elasticsearch clusters, tracing pipelines with Grafana Tempo and OpenTelemetry, and dashboards with Grafana, Kibana and Tempo.

Qualifications

• 7+ years in Site Reliability Engineering, Platform Engineering or DevOps.
• Hands-on experience administering Splunk Enterprise or Splunk Cloud.
• Strong knowledge of Splunk SPL.
• Experience with Elasticsearch/ELK, Prometheus, Grafana, Grafana Tempo, OpenTelemetry and Kafka.
• Experience implementing metrics, logs and traces as part of a modern observability strategy.
• Experience with Terraform and Infrastructure as Code.
• Programming experience in Python, Go, Ruby or Bash.

Responsibilities

• Design, deploy and operate enterprise observability platforms.
• Build and maintain Splunk Enterprise/Splunk Cloud infrastructure, including Indexers, Search Head Clusters, Heavy Forwarders and Deployment Servers.
• Deploy and operate large-scale Elasticsearch clusters for log analytics and search.
• Design, deploy and support distributed tracing platforms using Grafana Tempo and OpenTelemetry.
• Build and maintain end-to-end tracing pipelines, instrumentation standards and trace retention strategies.
• Scale Prometheus, Grafana, Kafka, Tempo and OpenTelemetry-based monitoring solutions.
• Develop dashboards, alerts, analytics and trace visualizations using Splunk SPL, Grafana, Kibana and Tempo.
• Automate infrastructure using Terraform and configuration management tools.

Skills

Splunk SPL

Terraform

Python

Go

Kubernetes

Prometheus

Grafana

OpenTelemetry

Kafka

Elasticsearch

Docker

CI/CD

AWS

Tools

Splunk Enterprise

Splunk Cloud

Elasticsearch

Kibana

Grafana

Tempo

OpenTelemetry

Terraform

Kubernetes

Docker

AWS

Azure

GCP

Ansible

Consul

Job description

Job Title: Lead Site Reliability Engineer – Observability
Location: Remote, United States
Work Arrangement: Fully remote within the United States
Employment Type: Contract (12 months)
Experience: 7+ years

About the Lead Site Reliability Engineer Role

Join an Observability team responsible for designing, building and operating enterprise platforms for logging, metrics, tracing and alerting across large-scale cloud infrastructure. You will lead initiatives that improve reliability, scalability and operational excellence.

Key Responsibilities

• Design, deploy and operate enterprise observability platforms.
• Build and maintain Splunk Enterprise/Splunk Cloud infrastructure, including Indexers, Search Head Clusters, Heavy Forwarders and Deployment Servers.
• Deploy and operate large-scale Elasticsearch clusters for log analytics and search.
• Design, deploy and support distributed tracing platforms using Grafana Tempo and OpenTelemetry.
• Build and maintain end-to-end tracing pipelines, instrumentation standards and trace retention strategies.
• Scale Prometheus, Grafana, Kafka, Tempo and OpenTelemetry-based monitoring solutions.
• Develop dashboards, alerts, analytics and trace visualizations using Splunk SPL, Grafana, Kibana and Tempo.
• Automate infrastructure using Terraform and configuration management tools.

Required Qualifications

• 7+ years in Site Reliability Engineering, Platform Engineering or DevOps.
• Hands-on experience administering Splunk Enterprise or Splunk Cloud.
• Strong knowledge of Splunk SPL.
• Experience with Elasticsearch/ELK, Prometheus, Grafana, Grafana Tempo, distributed tracing, OpenTelemetry and Kafka.
• Experience implementing metrics, logs and traces as part of a modern observability strategy.
• Experience with Terraform and Infrastructure as Code.
• Programming experience in Python, Go, Ruby or Bash.

Preferred Qualifications

• Splunk certification.
• Experience with Kubernetes, AWS/Azure/GCP, Ansible, Consul, CI/CD pipelines and service mesh technologies.
• Experience supporting FedRAMP or regulated environments.

Technology Stack

Splunk Enterprise, Splunk Cloud, Elasticsearch, ELK, Kibana, Prometheus, Grafana, Grafana Tempo, OpenTelemetry, Distributed Tracing, Kafka, Terraform, Kubernetes, Docker, Linux, Python, Go, Ruby, Bash, AWS, Ansible and Consul.

FedRAMP and IL-5 Environment Notice

The successful applicant may perform work in FedRAMP High or IL-5 environments.

Frequently Asked Questions About the Lead Site Reliability Engineer Role

1. What does a Lead Site Reliability Engineer do in this observability role?

You will design, deploy and operate enterprise platforms for logging, metrics, tracing and alerting across large-scale cloud infrastructure. You will also lead initiatives that improve reliability, scalability and operational excellence.

2. How much experience is required?

Candidates need 7+ years in Site Reliability Engineering, Platform Engineering or DevOps.

3. Is this role remote?

Yes. This is a fully remote position for candidates based in the United States.

4. Is this a full-time or contract position?

This is a 12-month contract position.

5. How important is Splunk experience?

It is central to the role. You should have hands‑on experience administering Splunk Enterprise or Splunk Cloud and strong knowledge of Splunk SPL.

6. Which Splunk components will I work with?

You will build and maintain Splunk infrastructure, including Indexers, Search Head Clusters, Heavy Forwarders and Deployment Servers.

7. What Elasticsearch work is involved?

You will deploy and operate large-scale Elasticsearch clusters for log analytics and search.

8. What distributed tracing tools are used?

The role focuses on Grafana Tempo and OpenTelemetry. You will build end-to-end tracing pipelines, instrumentation standards and trace retention strategies.

9. Which monitoring tools will I scale?

You will scale Prometheus, Grafana, Kafka, Tempo and OpenTelemetry‑based monitoring solutions.

10. Will I build dashboards and alerts?

Yes. You will develop dashboards, alerts, analytics and trace visualizations using Splunk SPL, Grafana, Kibana and Tempo.

11. Is Infrastructure as Code experience required?

Yes. Experience with Terraform and Infrastructure as Code is required, and you will automate infrastructure using Terraform and configuration management tools.

12. Which programming languages are expected?

Programming experience in Python, Go, Ruby or Bash is required.

13. What preferred qualifications will strengthen my application?

A Splunk certification, experience with Kubernetes, AWS/Azure/GCP, Ansible, Consul, CI/CD pipelines and service mesh technologies, and experience supporting FedRAMP or regulated environments.

14. Will this role involve FedRAMP or IL-5 environments?

Yes. The successful applicant may perform work in FedRAMP High or IL-5 environments.