Observability & AI Evaluation Engineer

NTT DATA, Inc.; Charlotte (NC); Hybrid; USD 97,000 - 145,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

401k with company match

Paid time off

Job summary

NTT DATA is seeking an Observability & Evaluation Engineer in Charlotte, NC to design and maintain telemetry, dashboards, and evaluation suites for production AI systems. You will ensure observability, reliability, and readiness across Tachyon agent releases, collaborating with platform engineers and product owners.

The role emphasizes building monitoring, evaluating prompt quality, latency, and safety signals, with an emphasis on runbooks and governance for release readiness.

Qualifications

• 7+ years of engineering experience with observability, monitoring, test automation, platform operations, or AI/ML systems.
• 5+ years of strong hands-on Python experience.
• 5+ years of Experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.
• 5+ years of Understanding of LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.
• 5+ years of Experience working in Agile engineering teams and production support environments.

Responsibilities

• Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
• Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
• Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
• Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
• Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
• Create runbooks and support documentation for priority agent releases.
• Analyze production behavior and recommend improvements to reliability, performance, and quality.

Skills

Python

Telemetry

Monitoring

Dashboards

SLOs

Alerts

LLM evaluation

Agile teams

Tools

Kubernetes

Cloud platforms

CI/CD

Job description

NTT DATA is seeking an Observability & Evaluation Engineer in Charlotte, NC to design and maintain telemetry, dashboards, and evaluation suites for production AI systems. You will ensure observability, reliability, and readiness across Tachyon agent releases, collaborating with platform engineers and product owners.

The role emphasizes building monitoring, evaluating prompt quality, latency, and safety signals, with an emphasis on runbooks and governance for release readiness.