Observability & Evaluation Engineer: Production AI

NTT Data Americas, Inc.; Charlotte (NC); On-site; USD 140,000 - 190,000; Full time

Job summary

NTT DATA in Charlotte, NC is seeking an Observability & Evaluation Engineer to build telemetry, tracing, dashboards, and evaluation suites for Tachyon agent releases, ensuring production AI systems are monitored and supported with clear operational visibility.

You will implement observability for LLM-powered applications, develop dashboards and alerts, define monitoring requirements with Product Owners, and automate evidence collection for release readiness.

Qualifications

• 7+ years of engineering experience in observability, monitoring, test automation, platform operations, or AI/ML systems.
• 5+ years of strong hands-on Python experience.
• 5+ years of experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.
• 5+ years of understanding LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.
• 5+ years of experience working in Agile engineering teams and production support environments.

Responsibilities

• Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
• Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
• Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
• Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
• Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
• Create runbooks and support documentation for priority agent releases.
• Analyze production behavior and recommend improvements to reliability, performance, and quality.

Skills

Python

Observability

Dashboards/Monitoring

LLM evaluation

Agile teams

Tools

Prometheus

OpenTelemetry

Kubernetes

CI/CD

Job description

NTT DATA in Charlotte, NC is seeking an Observability & Evaluation Engineer to build telemetry, tracing, dashboards, and evaluation suites for Tachyon agent releases, ensuring production AI systems are monitored and supported with clear operational visibility.

You will implement observability for LLM-powered applications, develop dashboards and alerts, define monitoring requirements with Product Owners, and automate evidence collection for release readiness.