AI Production Reliability & Observability Engineer

Rivago Infotech Inc; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Rivago Infotech Inc. seeks an AI DevOps/Observability Engineer to ensure reliability, performance, and readiness of Generative AI and LLM agent releases. You will bridge AI development and production operations with telemetry, evaluation pipelines, and comprehensive monitoring.

Responsibilities include deep tracing, dashboarding, alerting, SLOs, and automated evaluation suites to optimize cost, speed, and accuracy in production environments.

Qualifications

• Experience with framework-based evaluation tools (Ragas, DeepEval, TruLens) to measure hallucination, faithfulness, and relevancy.
• Proficiency with LLM tracing tools (LangSmith, LangFuse, Phoenix, Arize) and OpenTelemetry.
• Hands-on experience building production dashboards in Datadog, Prometheus, Grafana, or New Relic.
• Ability to define meaningful SLIs/SLOs and minimize alert fatigue.
• Strong background in integrating automated test frameworks into CI/CD for AI features.
• Analytical mindset to benchmark prompts, track regressions, and profile latency.

Responsibilities

• Observability & Telemetry: implement deep tracing and telemetry for LLM architectures and multi-agent workflows.
• Monitoring & Insights: design dashboards tracking system health, infra metrics, and AI performance indicators.
• Operational Readiness: establish alerting systems, define SLOs, curate runbooks, provide production readiness evidence.
• Evaluation & Testing Pipelines: develop automated continuous evaluation suites for LLM behavior and safety.
• Performance Analysis: analyze prompt efficiency, token usage, latency, and model performance.
• Production Operations: support deployment pipelines and incident management for live AI services.

Skills

LLM evaluation

Telemetry

Dashboards

SLOs

Test automation

Prompt analysis

Python

Cloud infra

DevOps

Tools

Datadog

Prometheus

Grafana

LangSmith

LangFuse

Phoenix

Arize

OpenTelemetry

LangChain

LlamaIndex

Job description

Rivago Infotech Inc. seeks an AI DevOps/Observability Engineer to ensure reliability, performance, and readiness of Generative AI and LLM agent releases. You will bridge AI development and production operations with telemetry, evaluation pipelines, and comprehensive monitoring.

Responsibilities include deep tracing, dashboarding, alerting, SLOs, and automated evaluation suites to optimize cost, speed, and accuracy in production environments.