AI Observability & Evaluation Engineer

Creative Solutions Services, LLC; Charlotte (NC); On-site; USD 120,000 - 190,000; Full time

Job summary

NTT DATA is seeking an Overwatch Observability & Evaluation Engineer to construct telemetry, dashboards, runbooks, and readiness evidence for Tachyon agent releases, ensuring production AI systems are monitored and evaluated.

You will implement observability for LLM-powered apps, build evaluation suites for agent behavior and prompt quality, and develop alerts, traces, and SLOs to improve reliability and performance across platforms.

Qualifications

• 7+ years of engineering experience in observability, monitoring, or AI/ML systems.
• Strong hands-on Python experience.
• Experience building dashboards, alerts, and service level objectives.

Responsibilities

• Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
• Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
• Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
• Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
• Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
• Create runbooks and support documentation for priority agent releases.
• Analyze production behavior and recommend improvements to reliability, performance, and quality.

Skills

Python

Observability

Monitoring

SLOs

Agile

Tools

OpenTelemetry

Kubernetes

CI/CD

Job description

NTT DATA is seeking an Overwatch Observability & Evaluation Engineer to construct telemetry, dashboards, runbooks, and readiness evidence for Tachyon agent releases, ensuring production AI systems are monitored and evaluated.

You will implement observability for LLM-powered apps, build evaluation suites for agent behavior and prompt quality, and develop alerts, traces, and SLOs to improve reliability and performance across platforms.