AI Observability & Evaluation Engineer

NTT DATA, Inc.; Charlotte (NC); On-site; USD 140,000 - 190,000; Full time

Job summary

NTT DATA, Inc. seeks an Overwatch Observability & Evaluation Engineer to build telemetry, dashboards, and evaluation suites for Tachyon agent releases. This role ensures production AI systems can be monitored, evaluated, improved, and supported with clear operational visibility.

You will design monitoring requirements, automate evidence collection, and create runbooks to enable rapid release readiness and governance in Agile teams.

Qualifications

• 7+ years of engineering experience with observability, monitoring, test automation, platform operations, or AI/ML systems.
• Experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.
• Understanding of LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.
• Experience working in Agile engineering teams and production support environments.

Responsibilities

• Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
• Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
• Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
• Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
• Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
• Create runbooks and support documentation for priority agent releases.
• Analyze production behavior and recommend improvements to reliability, performance, and quality.

Skills

Observability

Python

Telemetry

Dashboards

SLOs

Traces

Alerts

Monitoring

Agile

Production Ops

Tools

OpenTelemetry

Job description

NTT DATA, Inc. seeks an Overwatch Observability & Evaluation Engineer to build telemetry, dashboards, and evaluation suites for Tachyon agent releases. This role ensures production AI systems can be monitored, evaluated, improved, and supported with clear operational visibility.

You will design monitoring requirements, automate evidence collection, and create runbooks to enable rapid release readiness and governance in Agile teams.