AI Agent Observability & Evaluation Engineer

AMroute LLC; Bluffdale (UT); On-site; USD 130,000 - 170,000; Full time

Job summary

AMroute LLC in Charlotte, NC seeks an Overwatch Observability and Evaluation Engineer to implement telemetry, tracing, dashboards, and evaluation suites for high-priority AI agent releases.

You will define SLOs/SLIs, runbooks, and release-readiness evidence, collaborating with AI engineering, platform, DevOps, product, and risk teams to ensure production-grade observability and reliability.

Qualifications

• Bachelor's degree or higher in Computer Science, Data Science, AI/ML, Engineering, or related field.
• Experience implementing observability, evaluation, or monitoring for AI/ML, LLM, or agent-based systems.
• Strong Python scripting and automation skills.
• Ability to support production-grade AI releases with clear documentation, metrics, and operational readiness discipline.

Responsibilities

• Design and implement observability frameworks for LLM applications and AI agent workflows.
• Build telemetry, tracing, logging, metrics, and dashboarding capabilities for production AI systems.
• Develop evaluation suites to measure prompt quality, model behaviour, task success, groundedness, latency, reliability, and safety.
• Define and track SLOs, SLIs, service readiness criteria, and operational health indicators.
• Create automated test and evaluation workflows for model, prompt, and agent performance analysis.
• Build runbooks and operational readiness evidence for production releases.
• Partner with AI engineering, platform, DevOps, product, and risk teams to support priority agent launches.

Skills

LLM Evaluation

AI Agent Evaluation

Observability

Tracing

Telemetry

Metrics

Dashboards

SLOs

Runbooks

Test Automation

Prompt Performance Analysis

Model Performance Analysis

Python

Production Operations

Education

Bachelor's degree in Computer Science, Data Science, AI/ML, Engineering, or a related field

Tools

LangSmith

OpenTelemetry

Grafana

Prometheus

Datadog

CloudWatch

Azure Monitor

AI Reliability Engineering

LLMOps

MLOps

Incident Response

Release Readiness

Job description

AMroute LLC in Charlotte, NC seeks an Overwatch Observability and Evaluation Engineer to implement telemetry, tracing, dashboards, and evaluation suites for high-priority AI agent releases.

You will define SLOs/SLIs, runbooks, and release-readiness evidence, collaborating with AI engineering, platform, DevOps, product, and risk teams to ensure production-grade observability and reliability.