Senior Systems Engineer Onsite or Remote San Francisco, California

Evidently; San Francisco (CA); Hybrid; USD 200,000 - 250,000; Full time

Benefits offered by this job

Equity

US-based team

SF Bay Area office day

Job summary

Evidently in San Francisco is seeking a Senior Systems Engineer to own reliability for our production systems, including AI inference infra and data pipelines. You’ll define SLOs/SLIs, lead incident responses, and drive blameless postmortems to prevent fires and improve systems.

You’ll design scalable cloud infrastructure on GCP, manage IaC, CI/CD, observability, and security/compliance across HIPAA-like needs. Remote-friendly with core Pacific hours from the US.

Qualifications

• 8+ years of experience in SRE, infrastructure engineering, or DevOps.
• Defined and defended SLOs with error budgets and data-driven decisions.
• Led incident responses and postmortems to drive durable fixes.
• Strong experience with GCP (GKE, Cloud Run, AlloyDB, Pub/Sub) and CI/CD.

Responsibilities

• Own end-to-end reliability of production systems including AI inference pipelines.
• Define and own SLOs/SLIs and alerting dashboards.
• Lead incident response and blameless postmortems to prevent recurrence.
• Build and maintain scalable cloud infrastructure and IAM/security controls.
• Oversee data infrastructure health across GCP services and AI workloads.

Skills

SRE experience

Incidence management

GCP expertise

Python scripting

Shell scripting

Cloud infrastructure

Tools

GKE

Cloud Run

AlloyDB

BigQuery

Vertex AI

Terraform

Kubernetes

OpenTelemetry

GitHub Actions

Job description

At Evidently, we are raising the quality of care for every patient by empowering clinicians with the right information at the speed of thought. Our Clinical Data Intelligence platform extracts meaning from an ocean of clinical notes, outside records, and scanned documents — and delivers it to clinicians in the moment it matters. We're a small, mission-driven team building something genuinely hard and genuinely impactful. Learn more at evidently.com.

Who We're Looking For

We're looking for a Senior Systems Engineer who owns reliability the way a founding engineer owns their product: with a sense of personal accountability, eyes everywhere, and no tolerance for fires that were preventable.

This is a clear home-base role. Infrastructure and reliability are your domain and your primary responsibility. You are the person who notices before anyone else that something is trending wrong. You work alongside leadership as a partner in operational decision-making, not getting paged when things break, but instead making sure they don't break in the first place, and standing shoulder-to-shoulder with the team when they do. You bring a calm, systematic approach to incident response, help divide up the work intelligently under pressure, and run thorough postmortems that actually change behavior.

At the same time, you have the seniority and breadth to patrol adjacent territory. Integration work, security and compliance posture, database performance, AI inference infrastructure: you can cover these areas when they touch systems, and you're not waiting to be asked. You see the whole field.

We are an AI-first company, and that applies to how you work as much as what you build. We expect you to use AI tools aggressively, as an operator who directs them, not a passive user who accepts their output uncritically.

What You'll Do

Own Reliability

• Hold end-to-end accountability for the reliability, availability, and performance of Evidently's production systems: our SMART on FHIR platform, AI inference pipelines, data systems, and the integrations that connect them all.
• Define and own SLOs, SLIs, and error budgets for critical services; build the dashboards and alerting that make the health of the system visible to everyone, not just you.
• Lead incident response: triage fast, communicate clearly, coordinate the team, contain blast radius. Run blameless postmortems that actually result in durable fixes and systemic improvements, not just tickets.
• Proactively identify reliability risks before they become incidents, through capacity planning, load testing, chaos engineering, and systematic review of production signals.
• Build runbooks and on-call practices that make rotating on-call humane and effective, not a rotation people dread.

Build and Operate Infrastructure

• Design, implement, and maintain highly available, scalable, and secure cloud infrastructure on Google Cloud Platform: GKE, Cloud Run, AlloyDB, Cloud Storage, Pub/Sub, and adjacent services.
• Own Infrastructure as Code across all environments with a cloud-agnostic, open-minded, code-centric approach, applying standard software engineering discipline (version control, code review, testing, promotion gates).
• Build and maintain CI/CD pipelines (GitHub Actions) that make deployments fast, safe, and boring, with rollback, canary, and feature toggling patterns baked in.
• Manage observability infrastructure end-to-end: instrumentation, tracing, log pipelines, alerting, and dashboards structured around what actually matters operationally.
• Own data infrastructure health across GCP data systems, including AlloyDB, Cloud Storage, BigQuery, and Cloud Logging: performance tuning, query analysis, storage lifecycle management, connection management, backup/restore, and disaster recovery.
• Own AI and LLM inference infrastructure end-to-end: Vertex AI, Model Garden, and model serving platforms (vLLM, Cloud Run, GPU provisioning), latency SLOs, cost optimization, auto-scaling characteristics, and capacity planning for production AI workloads.

Set the Standard on Security and Compliance

• Maintain and improve our security posture across all infrastructure: network segmentation, IAM, secrets management, vulnerability management, audit logging.
• Own infrastructure-level compliance for HIPAA and SOC 2, not just checkbox compliance, but genuinely secure and auditable systems.
• Be a thought partner to engineering on security architecture decisions, particularly where AI workloads introduce new data handling surface area.

Patrol Adjacent Territory

• Cover integration infrastructure where it intersects with systems: EHR integrations, third-party APIs, data pipelines, and AI inference serving. You don't need to own these, but you understand them well enough to step in.
• Support backend engineers on runtime performance: Python async, FastAPI/Django serving characteristics, concurrency patterns, and where infrastructure choices constrain application-level performance.

What You'll Bring

• 8+ years of experience in SRE, infrastructure engineering, or DevOps, with a clear track record of owning reliability, not just contributing to it.
• You think in SLOs. You have defined them, defended them, and used error budgets to make real decisions about when to slow down feature work.
• Deep experience with incident management: you've led responses to serious production incidents, written the postmortems, and driven the follow-through.
• Expert-level command of GCP (GKE, Cloud Run, Cloud SQL/AlloyDB, Pub/Sub, IAM, VPC, Cloud Armor) or directly comparable experience on AWS or Azure with a credible GCP migration story.
• Very strong shell scripting skills (Bash) and fluency in Python for automation; comfortable reading application code well enough to understand its operational behavior.

Depth That Matters

• Observability done right: you've built meaningful monitoring from the ground up, not just enabled a tool. You know the difference between dashboards that look good and dashboards that catch problems.
• Deep knowledge of GCP data infrastructure and database operational characteristics: query planning and tuning in PostgreSQL/AlloyDB/BigQuery, log management and retention in Cloud Logging, data lifecycle management in Cloud Storage, analytical workloads in BigQuery, and how to diagnose issues across these services as they scale.
• Understanding of container and Kubernetes internals well beyond YAML: scheduling, resource management, network policies, pod disruption budgets, autoscaling behavior.
• Security and compliance grounding: you've operated in regulated environments (HIPAA, SOC 2, or similar) and understand the difference between compliance theater and systems that are actually secure.
• Deep operational mastery of AI and LLM inference serving, including Vertex AI and Model Garden: GPU provisioning and orchestration, model deployment strategies, optimizing inference latency and throughput, GPU cost controls, and managing the unique reliability patterns of production LLM workloads at scale.

Mindset

• You have a strong operational instinct. You notice anomalies, you ask why things are trending the way they are, and you're not satisfied until you understand the root cause.
• You treat reliability as a product you own, not a set of tasks assigned to you.
• You're calm under pressure, structured in how you communicate during incidents, and rigorous in follow-through after them.
• You're senior enough to have strong opinions about how things should be done and humble enough to change them when presented with better evidence.
• You use AI tools fluently as a practitioner, writing infrastructure code, debugging incidents, drafting runbooks, with the judgment to know when the output is right and when it needs correction.

Bonus

• Familiarity with healthcare data standards: FHIR, HL7, SMART on FHIR application architecture.
• Experience operating infrastructure for LLM or AI inference workloads at production scale.
• Background in building internal developer platforms or golden-path tooling that makes other engineers more effective.
• Strong hands-on experience or familiarity with Infrastructure as Code tooling such as Kubernetes, Terraform, or Pulumi.
• Experience with OpenTelemetry for instrumentation and distributed tracing across complex environments.

Who You'll Work With

You'll join a small, highly effective, self-driven engineering team where resourcefulness is prized and engineers operate with a high degree of autonomy. You will serve as a peer to the CTO and technical leadership—not as a support function, but as a co-owner of how the platform runs. You'll work closely with backend, data, and AI/ML engineers on the systems they build and depend on, and directly with leadership on operational priorities and incident response. The team is concentrated in the Bay Area with a weekly San Francisco office day; remote candidates are welcome, though you must be able to work core Pacific hours and must be based in the US.

Compensation: $200k – $250k + equity

We’re a team on a mission to elevate the quality of care for every patient, by empowering clinicians with the right information at the speed of thought. Learn more about us.