Senior AI Observability Engineer – Agentic Systems

Nvidia Corporation in; Santa Clara (CA); On-site; USD 168,000 - 270,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA is seeking a Senior Software Engineer for Agentic AI and Observability (Finance) to join the BizApps SRE team. You will build observability for agentic AI applications in production and help ensure reliable, scalable AI-powered services across the enterprise.

This role emphasizes deploying pipelines, defining SLOs/SLIs, instrumenting LLM workflows, and driving incident response. You will partner with platform, data, and application teams to close gaps and improve reliability throughout

Qualifications

• BS or MS in Computer Science, Software Engineering, or a related field (or equivalent experience).
• 8+ years of experience with strong software engineering skills in Python and experience with modern CI/CD practices.
• Hands-on experience with container orchestration (Kubernetes, Docker) and cloud infrastructure.
• Experience with observability and monitoring platforms such as Datadog, OpenTelemetry, Grafana, or Prometheus.
• SRE / DevOps approach—taking ownership of production systems, automating toil, and building for reliability.
• Excellent problem-solving skills and a proven track record of debugging complex distributed systems.

Responsibilities

• Build and implement observability solutions for agentic AI applications in production, including metrics, tracing, logging, and alerting.
• Develop and sustain deployment pipelines and reliability tools for the Agentic AI Factory model.
• Partner with AI application teams to define SLOs/SLIs and ensure production readiness for new agent deployments.
• Instrument LLM-based workflows for performance, cost, and quality monitoring, covering multi-step reasoning chains, token usage, and tool orchestration.
• Drive incident response, root cause analysis, and reliability improvements for BizApps AI services.
• Identify and close observability gaps outstanding to agentic AI systems, such as hallucination detection and orchestration failure tracing.
• Work jointly with platform, data, and application engineering groups to integrate reliability throughout the development lifecycle.

Skills

Python

CI/CD

Distributed systems debugging

Problem solving

SRE/DevOps mindset

Production ownership

Education

BS/MS in Computer Science or related field

Tools

Kubernetes

Docker

Datadog

OpenTelemetry

Grafana

Prometheus

Job description

NVIDIA is seeking a Senior Software Engineer for Agentic AI and Observability (Finance) to join the BizApps SRE team. You will build observability for agentic AI applications in production and help ensure reliable, scalable AI-powered services across the enterprise.

This role emphasizes deploying pipelines, defining SLOs/SLIs, instrumenting LLM workflows, and driving incident response. You will partner with platform, data, and application teams to close gaps and improve reliability throughout