Senior Site Reliability Engineer - Observability

Okta Inc.; Washington (IL); Hybrid; USD 194,000 - 267,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Vision insurance

401(k)

Flexible spending account

Paid leave

Job summary

Okta is seeking a Staff Observability Site Reliability Engineer in Bellevue, WA to own and evolve our Splunk ecosystem, treating infrastructure as code with Terraform and coding in Go, Python, or Ruby.

You will design scalable observability, optimize Splunk Cloud across 1000+ services, participate in on-call, automate deployments, and build intuitive dashboards for cross-source data.

Qualifications

• 5+ years in SRE/DevOps or systems engineering focused on high-availability systems.
• Experience scaling Splunk Cloud across 1000+ services.
• Strong coding in Go, Python, or Ruby to automate workflows.
• Experience with Terraform for IaC and Kubernetes orchestration.
• Familiarity with Linux internals and networking.

Responsibilities

• Own and evolve the Splunk observability platform for reliability.
• Automate deployment and scaling of observability agents.
• Lead on-call rotations and post-incident reviews.
• Develop dashboards and visualizations to correlate data.
• Collaborate with SRE teams and business partners.

Skills

Splunk

Terraform

Go

Python

Ruby

Linux

Kubernetes

SRE/DevOps

Tools

Splunk Cloud

HEC

OpenTelemetry

Job description

# Staff Site Reliability Engineer - ObservabilityBellevue, Washington**Secure Every Identity, from AI to Human**Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk.### **Position Overview:**We are seeking a highly technical **Staff** **Observability** **Site Reliability Engineer** with a specialty in Splunk to own and evolve our Splunk ecosystem. In this role, you will move beyond simple monitoring to delivering a world class, comprehensive, scalable Observability Platform that enables our SRE teams and business partners. You will treat **infrastructure as code**—utilizing Terraform and strong coding proficiency in **Go, Python, or Ruby**—to automate the deployment of agents and collectors across complex distributed systems.**Key Responsibilities*** Automated Infrastructure: Design, build, and maintain scalable observability infrastructure using tools like Terraform.* Splunk Engineering: Optimize the collection, processing, and storage of log data to ensure high reliability and low latency of our Splunk services* Incident Response: Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and "observability-driven development."* Automation: Eliminate "toil" by automating the deployment and scaling of observability agents and collectors.**Required Skills & Experience (The Essentials)****Log Management: Minimum 5+ Experience scaling and managing Splunk Cloud at scale (1000+ SVCs), including Workload Management (WLM) and HEC optimization.** **Visualization: Expertise in creating intuitive, actionable Splunk dashboards that correlate data across multiple sources.****SRE Mindset: Minimum 5+ years of experience in an SRE, DevOps, or Systems Engineering role with a focus on high-availability systems.*** **Programming Proficiency:** Strong coding skills in **SPL**, **Go** for building internal tools and automating workflows.* **Distributed Systems:** Deep understanding of Linux internals, networking (TCP/IP, DNS, Load Balancing), and container orchestration (Kubernetes/EKS).* **Problem Solving:** A data-driven approach to debugging complex, cross-service performance bottlenecks.**Bonus Skills (The "Nice-to-Haves")*** **Telemetry Standards:** Hands-on experience with OpenTelemetry (OTel), Vector, or similar frameworks for instrumenting applications.* **Charge-back app:** Experience in implementing Splunk charge-back app for usage reporting**Cloud Platforms:** Experience managing observability native tools within AWS or GCP.**Additional requirements:*** This position requires the ability to access federal environments and/or have access to protected federal data. As a condition of employment for this position, the successful candidate must be able to submit documentation establishing U.S. Person status (e.g. a U.S. Citizen, National, Lawful Permanent Resident, Refugee, or Asylee. 22 CFR 120.15) upon hire.* This person must attend in person onboarding in our San Francisco office the first week of employment.#LI-MM#LI-Hybrid P14596\_3372199Below is the annual base salary range for candidates located in California (excluding San Francisco Bay Area), Colorado, Illinois, New York and Washington. Your actual base salary will depend on factors such as your skills, qualifications, experience, and work location. In addition, Okta offers equity (where applicable), bonus, and benefits, including health, dental and vision insurance, 401(k), flexible spending account, and paid leave (including PTO and parental leave) in accordance with our applicable plans and policies. To learn more about our Total Rewards program please visit: https://rewards.okta.com/us.The annual base salary range for this position for candidates located in California (excluding San Francisco Bay Area), Colorado, Illinois, New York, and Washington is between:$194,000—$267,000 USD**The Okta Experience*** Supporting Your Well-Being* Driving Social Impact* Developing Talent and Fostering Connection + CommunityWe are intentional about connection. Our global community, spanning over 20 offices worldwide, is united by a drive to innovate. Your journey begins with an immersive, in-person onboarding experience designed to accelerate your impact and connect you to our mission and team from day one. Okta is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, ancestry, marital status, age, physical or mental disability, or status as a protected veteran. We also consider for employment qualified applicants with arrest and convictions records, consistent with applicable laws. If reasonable accommodation is needed to complete any part of the job application, interview process, or onboarding please use this Form to request an accommodation. Notice for New York City Applicants & Employees: Okta may use Automated Employment Decision Tools (AEDT), as defined by New York City Local Law 144, that use artificial intelligence, machine learning, or other automated processes to assist in our recruitment and hiring process. In accordance with NYC Local Law 144, if you are an applicant or employee residing in New York City, please click here to view our full NYC AEDT Notice. Okta is committed to complying with applicable data privacy and security laws and regulations. For more information, please see our Personnel and Job Candidate Privacy Notice at https://www.okta.com/legal/personnel-policy/.