Staff Software Engineer, Observability/Site Reliability

LinuxCareers; Palo Alto (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

PsiQuantum seeks an experienced Staff-level Observability Engineer to design and operate fault-tolerant telemetry across on-prem Kubernetes and cloud. You will own metrics, logs, and traces collection, storage, and dashboards, while mentoring engineers and guiding standards.

Strong production OpenTelemetry, Prometheus, Grafana, and Linux skills are essential. You will collaborate with hardware teams, instrument custom devices, and participate in on-call rotations and post-incident reviews,

Qualifications

• 8+ years building and operating observability systems in production.
• Degree in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, Physics, or closely related field.
• Deep hands-on Prometheus and Grafana at scale.
• Production OpenTelemetry experience including Collector and OTLP.
• Hands-on with self-managed on-prem Kubernetes and cloud, IaC tools.
• Strong Python and at least one of Go or Rust for exporters and tooling.
• Solid Linux and networking fundamentals.
• Experience carrying pager, incident response, and post-incident reviews.

Responsibilities

• Design, deploy, and operate an observability platform across on-prem Kubernetes and cloud.
• Roll out OpenTelemetry across fleet; configure collectors and telemetry pipelines.
• Run Prometheus in production with service discovery, rules, Alertmanager, and long-term storage.
• Instrument hardware and build exporters for device telemetry.
• Publish telemetry standards for metrics naming, cardinality budgets, and tracing context.
• Participate in on-call rotation, diagnose incidents, and lead blameless postmortems.
• Mentor engineers, lead design reviews, and align observability needs across teams.
• Leverage AI/LLM tools to accelerate engineering tasks while ensuring security and code quality.

Skills

Observability

Prometheus

Grafana

OpenTelemetry

Python

Go

Rust

Linux fundamentals

Incident response

Leadership

Education

Bachelor's or Master's in CS/CE/EE/Math/Physics

Tools

Kubernetes

Terraform

OpenTelemetry

Prometheus

Grafana

Job description

PsiQuantum is building fault-tolerant quantum computers using silicon photonics. The Observability team builds and operates the telemetry platform that monitors quantum computing systems, on-premises Kubernetes clusters, cloud infrastructure, and custom hardware including control electronics and cryostats. This Staff-level role combines hands-on observability engineering with technical leadership, owning major parts of the stack from design through operation.

What You'll Do

• Design, deploy, and operate observability platform across on-premises Kubernetes and cloud, covering collection, storage, querying, visualization, and alerting for metrics, logs, and traces
• Roll out OpenTelemetry across the fleet: deploy and configure Collectors, design telemetry pipelines, and help service teams adopt OTel SDKs in Python, Rust, and other languages
• Run Prometheus in production including service discovery, scrape configuration, recording and alerting rules, Alertmanager routing, retention, high availability, and long-term storage
• Work with hardware teams to instrument custom devices, control electronics, and cryostats; build exporters and collectors for device telemetry
• Define and publish telemetry standards for metric naming, cardinality budgets, structured logging, and trace context propagation
• Participate in on-call rotation, respond to incidents, diagnose production issues, and lead blameless post-incident reviews
• Mentor engineers, lead design reviews, and coordinate observability needs across software, hardware, research, and operations teams
• Use AI and LLM tools responsibly to accelerate engineering work including code generation, query building, and incident analysis

What You Need

• 8+ years of professional software, infrastructure, or site reliability engineering experience, including significant time building and operating observability systems in production
• Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, Physics, or closely related technical discipline
• Deep hands-on experience with Prometheus and Grafana, including PromQL, recording and alerting rules, Alertmanager, exporters, and dashboard design at scale
• Production experience with OpenTelemetry, including Collector, OTLP, SDK, automatic instrumentation, and semantic conventions
• Hands-on experience running workloads on self-managed on-premises Kubernetes and cloud with containers, Helm, and infrastructure as code such as Terraform
• Strong programming skills in Python and at least one of Go or Rust, with ability to write production-quality exporters, Collector components, and automation
• Solid knowledge of Linux and networking fundamentals including processes, filesystems, resource limits, DNS, TCP/IP, and TLS
• Experience carrying a pager for production systems, leading incident response, running post-incident reviews, and designing actionable alerts
• Staff-level technical leadership: leading design work, setting standards, mentoring engineers, influencing across teams, and communicating tradeoffs
• Proficiency with modern AI/LLM engineering tools such as coding copilots, model APIs, and retrieval workflows, with disciplined approach to security and code review

Nice to Have

• Experience with scalable metrics, log, and trace backends such as Thanos, Mimir, VictoriaMetrics, Loki, Tempo, OpenSearch, or ClickHouse
• Experience collecting telemetry from laboratory or industrial equipment over protocols such as Modbus, OPC UA, SNMP, SCPI, or serial interfaces
• Familiarity with cryogenic, vacuum, laser, or other lab systems, or with software used in laboratories, data centers, or advanced manufacturing
• Experience with high-frequency or high-cardinality time-series data, including downsampling and data lifecycle management
• Experience running observability across multiple sites with hybrid on-premises and cloud environments including constrained network connectivity
• Experience with eBPF-based observability, continuous profiling, or Kubernetes operators
• Experience with SLO tooling, synthetic monitoring, or incident management platforms such as PagerDuty
• Familiarity with Kafka, AWS, or VAST Data
• Contributions to OpenTelemetry, Prometheus, Grafana, or related open-source projects
• Background in physics or quantum computing is helpful but not required