Staff Software Engineer, Observability/Site Reliability

PsiQuantum; United States; Remote; USD 180,000 - 260,000; Full time

Job summary

PsiQuantum in Palo Alto, CA (remote) is seeking a Staff Software Engineer for the Observability team. You will wire up OpenTelemetry instrumentation, scale Prometheus and Grafana, and turn raw signals into trusted dashboards and alerts that guide production decisions.

This hands-on role combines design and operation across on-premises Kubernetes, cloud, and hardware-in-the-loop systems. You will lead architecture, own major components of the observability stack, participate in on-call rotation,

Qualifications

• 8+ years building and operating observability systems in production.
• Bachelor's or Master's degree in CS/Engineering/Physics or related.
• Deep Prometheus and Grafana experience at scale.
• Production OpenTelemetry including Collector, OTLP, and instrumentation.
• Kubernetes on-prem and cloud with Terraform; GitLab CI preferred.
• Python and Go or Rust; production exporters and collectors.
• Linux, networking basics; debugging distributed systems.
• Experience on-call and post-incident reviews.
• Staff-level technical leadership and cross-team influence.
• Familiar with AI/LLM tooling, privacy, and secure coding practices.

Responsibilities

• Design, deploy, and operate the observability platform across on-premises and cloud environments.
• Roll out OpenTelemetry across the fleet and guide service teams in instrumentation.
• Run Prometheus in production, including discovery, rules, Alertmanager, and storage.

Skills

Observability

Prometheus

Grafana

OpenTelemetry

Python

Go

Rust

Kubernetes

Linux

Incident response

Leadership

AI/LLM tooling

Education

Bachelor's or Master's degree in CS/Engineering/Physics or related

Tools

Terraform

GitLab CI

OTLP

Job description

Staff Software Engineer, Observability/Site Reliability at PsiQuantum, Palo Alto, California (remote). A lead-level full time position in the 5100 - Electronics team, listed under software engineering.

Come join us.

Job Summary

The Observability team builds and runs the telemetry platform that lets PsiQuantum see how its quantum computing systems, and the infrastructure behind them, are behaving. We collect metrics, logs, and traces from on-premises Kubernetes clusters, cloud infrastructure, and the custom hardware that makes up a photonic quantum computer, including control electronics and cryostats. Engineering, operations, hardware, and research teams rely on this data to run the machine, catch problems early, and work out what went wrong when something fails.

We are looking for a Staff Software Engineer to join the Observability team. You will wire up OpenTelemetry instrumentation and collection, run and scale Prometheus and Grafana, and turn raw signals into dashboards and alerts that people trust. A large part of the job is working with hardware teams: deciding what their devices should report, and getting that data off the device and into the platform reliably.

This is a hands-on role with Staff-level scope. You will own major parts of the observability stack from design through operation, share the team's on-call rotation, and set the conventions other teams follow when they instrument their own systems. You will combine hands-on delivery with technical leadership, sound judgment under ambiguity, and a strong bias toward simple, maintainable systems.

Responsibilities

• Design, deploy, and operate the observability platform across on-premises Kubernetes and cloud, covering collection, storage, querying, visualization, and alerting for metrics, logs, and traces.
• Roll out OpenTelemetry across the fleet: deploy and configure Collectors, design telemetry pipelines, and help service teams adopt OTel SDKs and instrumentation in Python, Rust, and other languages.
• Run Prometheus in production, including service discovery, scrape configuration, recording and alerting rules, Alertmanager routing, retention, high availability, and long-term storage.

Requirements

• 8+ years of professional software, infrastructure, or site reliability engineering experience, including significant time building and operating observability systems in production.
• A Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, Physics, or a closely related technical discipline is required.
• Deep hands-on experience with Prometheus and Grafana, including PromQL, recording and alerting rules, Alertmanager, exporters, dashboard design, and running both reliably at scale.
• Production experience with OpenTelemetry, including the Collector, OTLP, SDK and automatic instrumentation, and semantic conventions.
• Hands-on experience running workloads on self-managed, on-premises Kubernetes and in the cloud, with containers, Helm, and infrastructure as code such as Terraform; practical experience with GitLab and GitLab CI is strongly preferred.
• Strong programming skills in Python and at least one of Go or Rust, with the ability to write production-quality exporters, Collector components, and automation rather than only configuring off-the-shelf tools.
• Solid knowledge of Linux and networking fundamentals, including processes, filesystems, resource limits, DNS, TCP/IP, and TLS, and experience debugging distributed systems from the host up.
• Experience carrying a pager for production systems, leading incident response, running post-incident reviews, and designing alerts people act on.
• Staff-level technical leadership: leading design work, setting standards other teams adopt, mentoring engineers, influencing across teams, and communicating tradeoffs to technical and non-technical stakeholders.
• Proficiency with modern AI/LLM engineering tools, such as coding copilots and agents, model APIs, and retrieval workflows; a disciplined approach to privacy, security, hallucination risk, and generated-code review.

Nice to Have

• Experience with scalable metrics, log, and trace backends such as Thanos, Mimir, VictoriaMetrics, Loki, Tempo, OpenSearch, or ClickHouse.
• Experience collecting telemetry from laboratory or industrial equipment over protocols such as Modbus, OPC UA, SNMP, SCPI, or serial interfaces, or through vendor SDKs.
• Familiarity with cryogenic, vacuum, laser, or other lab systems, or with software used in laboratories, data centers, or advanced manufacturing.
• Experience with high-frequency or high-cardinality time-series data, including downsampling, aggregation, and data lifecycle management.
• Experience running observability across multiple sites and hybrid on-premises and cloud environments, including sites with constrained network connectivity.
• Experience with eBPF-based observability, continuous profiling, or Kubernetes operators.