Staff Software Engineer, Observability/Site Reliability

PSI Quantum; Palo Alto (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

PsiQuantum is seeking a Staff Software Engineer for the Observability team to own and evolve the telemetry platform across on‑prem Kubernetes and cloud. You will wire up OpenTelemetry, scale Prometheus and Grafana, and translate raw signals into trusted dashboards and alerts.

You will collaborate with hardware teams to instrument devices and ensure reliable data flow into the platform. You will lead design reviews, mentor engineers, and help set conventions for instrumenting systems, while

Qualifications

• Staff-level observability experience with ownership of stack design and operation.
• Hands-on instrumentation of devices and telemetry in Kubernetes and cloud environments.
• Ability to collaborate with hardware teams to collect reliable telemetry data.

Responsibilities

• Design, deploy, and operate the observability platform across on‑prem Kubernetes and cloud.
• Roll out OpenTelemetry, design telemetry pipelines, and guide SDK instrumentation in multiple languages.
• Run Prometheus in production: service discovery, rules, alerts, and long‑term storage.
• Manage Grafana dashboards and data sources as code across environments.
• Instrument custom devices and lab equipment to feed telemetry into the platform.
• Define telemetry standards for naming, cardinality, logging, and tracing.
• Partner with service owners to define SLOs and actionable alerts; reduce noise.
• Lead on‑call rotations and post‑incident reviews; drive fixes and improvements.
• Operate the platform as a production service with CI/CD and infrastructure as code.
• Apply AI/LLM tooling responsibly to accelerate engineering while maintaining security.

Skills

OpenTelemetry

Prometheus

Grafana

Python

Rust

Kubernetes

GitLab CI/CD

SLOs

Incident response

LLM tooling

Tools

OpenTelemetry

Prometheus

Grafana

Python

Rust

Kubernetes

GitLab CI/CD

Job description

PsiQuantum'smission is to build the first useful quantum computers-machines capable of delivering the breakthroughs the field has long promised. Since our founding in 2016, our singular focus has been to build and deploy million-qubit, fault-tolerant quantum systems.

Quantum computers harness the laws of quantum mechanics to solve problems that even the most advanced supercomputers or AI systems will never reach. Their impact will span energy, pharmaceuticals, finance, agriculture, transportation, materials, and other foundational industries.

Our architecture and approachisbased on silicon photonics. Byleveragingthe advanced semiconductor manufacturing industry-including partners like GlobalFoundries-we use the same high-volume processes that already produce billions of chips for telecom and consumer electronics. Photonics offers natural advantages for scale: photonsdon'tfeel heat, are immune to electromagnetic interference, and integrate with existing cryogenic cooling and standard fiber-optic infrastructure.

In 2024,PsiQuantumannounced government-funded projects to support the build-out of our first utility-scale quantum computers in Brisbane, Australia, and Chicago, Illinois. These initiatives reflect a growing recognition that quantum computing will be strategically and economically defining-and that now is the time to scale.

PsiQuantumalso develops the algorithms and software needed to make these systems commercially valuable. Our application, software, and industry teams work directly with leading Fortune 500 companies-including Lockheed Martin, Mercedes-Benz, Boehringer Ingelheim, and Mitsubishi Chemical-to prepare quantum solutions for real world impact.

Quantum computing is not an extension of classical computing. Itrepresentsa fundamental shift-and a path to mastering challenges that cannot besolvedany other way. The potential is enormous, and we have a clearpathto make it real.

Come join us.

Job Summary

The Observability team builds and runs the telemetry platform that lets PsiQuantum see how its quantum computing systems, and the infrastructure behind them, are behaving. We collect metrics, logs, and traces from on-premises Kubernetes clusters, cloud infrastructure, and the custom hardware that makes up a photonic quantum computer, including control electronics and cryostats. Engineering, operations, hardware, and research teams rely on this data to run the machine, catch problems early, and work out what went wrong when something fails.

We are looking for a Staff Software Engineer to join the Observability team. You will wire up OpenTelemetry instrumentation and collection, run and scale Prometheus and Grafana, and turn raw signals into dashboards and alerts that people trust. A large part of the job is working with hardware teams: deciding what their devices should report, and getting that data off the device and into the platform reliably.

This is a hands‑on role with Staff‑level scope. You will own major parts of the observability stack from design through operation, share the team's on‑call rotation, and set the conventions other teams follow when they instrument their own systems. You will combine hands‑on delivery with technical leadership, sound judgment under ambiguity, and a strong bias toward simple, maintainable systems.

Responsibilities

• Design, deploy, and operate the observability platform across on‑premises Kubernetes and cloud, covering collection, storage, querying, visualization, and alerting for metrics, logs, and traces.
• Roll out OpenTelemetry across the fleet: deploy and configure Collectors, design telemetry pipelines, and help service teams adopt OTel SDKs and instrumentation in Python, Rust, and other languages.
• Run Prometheus in production, including service discovery, scrape configuration, recording and alerting rules, Alertmanager routing, retention, high availability, and long‑term storage.
• Manage Grafana dashboards, alerts, and data sources as code so they are versioned, reviewed, and reproducible across environments.
• Work with hardware teams to instrument custom devices, control electronics, cryostats, and other lab equipment; build the exporters and collectors that bring device telemetry into the platform, including from equipment that was never designed to be monitored.
• Define and publish telemetry standards - metric and label naming, cardinality budgets, structured logging, trace context propagation, and resource attributes - so data from different teams and systems can be correlated.
• Partner with service owners and operations teams to define SLOs and alerts that reflect real impact, reduce alert noise, and make every page actionable.
• Take part in the team's on‑call rotation: respond to incidents affecting the observability platform, help other teams diagnose production issues with telemetry, and lead blameless post‑incident reviews through to completed fixes.
• Run the platform as a production service in its own right, with capacity planning, cost management, upgrades, backup and recovery, and monitoring of the monitoring stack itself, delivered through GitLab CI/CD and infrastructure as code.
• Use AI and LLM tooling to accelerate engineering and operational work responsibly - including coding assistants, query and dashboard generation, and incident analysis - while applying rigorous review, security, and validation practices.
• Work with the team's technical lead on architecture and roadmap; lead design reviews, mentor engineers, document consequential decisions, and coordinate with software, hardware, research, infrastructure, and operations teams to turn their observability needs into a shared plan.

Daily Functions

• Instrument and collect: Configure OTel Collectors and Prometheus scrape targets, write exporters, and help teams instrument new services and devices.
• Build and review: Write code and configuration, review merge requests for platform and dashboard changes, and simplify pipelines that are hard to understand or operate.
• Work with hardware teams: Learn how a new device or cryostat behaves, agree on what to measure and alert on, and get its data flowing.
• Watch and respond: Check platform health and alert quality, handle on‑call pages, and help other teams debug with metrics, logs, and traces.
• Shape the platform: Work through storage, retention, cost, and tooling tradeoffs with the technical lead; unblock other engineers; keep near‑term work aligned with the roadmap.
• Apply AI with judgment: Use LLM‑assisted tools for queries, configuration, code, and documentation, then verify outputs to the standard expected for production infrastructure.

Experience