AI Systems Observability Engineer — Evaluation & Readiness

NTT DATA North America; Charlotte (NC); On-site; USD 130,000 - 185,000; Full time

Job summary

NTT DATA North America in Charlotte, NC seeks an Observability & Evaluation Engineer to design and implement telemetry, tracing, dashboards, and readiness evidence for AI-powered platforms. You will collaborate with platform teams to define monitoring requirements, build evaluation suites for agent behavior and retrieval quality, and automate evidence for release readiness and governance.

This role emphasizes reliability, performance, and quality across production systems and requires strong

Qualifications

• 7+ years in observability, monitoring, or AI/ML system engineering.
• 5+ years Python hands-on experience.
• 5+ years building dashboards, metrics, alerts, and SLOs in production.
• 5+ years understanding of LLM evaluation and AI quality assessment.
• 5+ years Agile engineering and production support.

Responsibilities

• Implement observability and telemetry for LLM-powered apps and services.
• Build evaluation suites for agent behavior and performance.
• Develop dashboards, alerts, traces, metrics, and readiness reporting.
• Collaborate with platform engineers to define monitoring and evaluation requirements.
• Automate evidence collection for releases and governance.
• Create runbooks and support docs for agent releases.
• Analyze production behavior to improve reliability and quality.

Skills

Observability

Python

Dashboards

SLOs & alerts

AI/ML systems

Agile

Tools

OpenTelemetry

Kubernetes

CI/CD

Job description

NTT DATA North America in Charlotte, NC seeks an Observability & Evaluation Engineer to design and implement telemetry, tracing, dashboards, and readiness evidence for AI-powered platforms. You will collaborate with platform teams to define monitoring requirements, build evaluation suites for agent behavior and retrieval quality, and automate evidence for release readiness and governance.

This role emphasizes reliability, performance, and quality across production systems and requires strong