Senior AI Reliability Engineer - Observability (Hybrid)

Cisco; San Jose (CA); Hybrid; USD 168,000 - 245,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401(k) with match

Paid parental leave

Disability coverage

Life insurance

RSU grants

Job summary

Cisco's Enterprise AI team in San Jose, CA or North Carolina is seeking a Senior Software Engineer focused on application reliability for AI-powered applications. You will own feature uptime, observability, and resilience, collaborating with developers, data engineers, and SREs to keep APIs and AI services healthy.

You will build LangGraph-based agents, develop Looker dashboards on BigQuery/BigTable, and write complex SQL for operational analytics.

Qualifications

• 7+ years of software engineering focusing on reliability or production operations.
• Strong Python development with production tooling and automation
• Production GCP experience with GKE, BigQuery, and BigTable
• Experience designing and operating SLI/SLO frameworks and burn-rate alerting
• Strong debugging with distributed tracing and structured logs

Responsibilities

• Define, implement, and enforce feature SLIs, SLOs, and error budgets for APIs, RAG systems, AI agents, and user-facing applications.
• Build Looker dashboards on BigQuery/BigTable for real-time visibility into feature health and usage patterns.
• Design LangGraph-based agents for automated issue identification and remediation: anomaly detection on BQ logs, root-cause diagnosis, auto-rollback, feature flag kills, and self-healing workflows.
• Develop agent evaluation harnesses to benchmark agent performance and regression test evolving agents.
• Write complex SQL (BigQuery) for usage trends, anomaly detection, and operational analytics; design BQ table schemas for observability.
• Analyze application usage trends to identify reliability risks and capacity needs.
• Partner with development teams to embed reliability practices into deployment safety, structured logging, and tracing.
• Lead incident response, root-cause analysis, and blameless postmortems focused on feature impact.
• Build Python-based tooling to reduce MTTD and MTTR for application issues.
• Stay current with AI landscape and apply new techniques to improve platform reliability.

Skills

Python

Observability

Reliability engineering

SQL (BigQuery)

GCP

Education

Bachelor's or Master's in CS/Engineering

Tools

GKE (Kubernetes)

BigQuery

BigTable

Looker

LangGraph

Job description

Cisco's Enterprise AI team in San Jose, CA or North Carolina is seeking a Senior Software Engineer focused on application reliability for AI-powered applications. You will own feature uptime, observability, and resilience, collaborating with developers, data engineers, and SREs to keep APIs and AI services healthy.

You will build LangGraph-based agents, develop Looker dashboards on BigQuery/BigTable, and write complex SQL for operational analytics.