AI Observability Engineer: Evaluation and Readiness

Creative Solutions Services, LLC; Charlotte (NC); On-site; USD 140,000 - 200,000; Full time

Job summary

NTT DATA in Charlotte, NC seeks an Observability & Evaluation Engineer to build telemetry, dashboards, and evaluation suites for Tachyon agent releases. You will ensure AI systems are monitored, evaluated, and supported with clear operational visibility.

The role emphasizes developing runbooks, alerts, SLOs, and readiness evidence, collaborating with platform engineers and product owners, and automating evidence collection for release readiness and governance.

Qualifications

• 7+ years in observability, monitoring, or AI/ML systems.
• 5+ years Python experience.
• 5+ years with dashboards, metrics, alerts, and production monitoring.
• 5+ years of LLM evaluation or AI quality assessment.
• 5+ years in Agile teams and production support.

Responsibilities

• Implement observability and telemetry for LLM-powered applications.
• Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
• Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
• Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
• Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
• Create runbooks and support documentation for priority agent releases.
• Analyze production behavior and recommend improvements to reliability, performance, and quality.

Skills

Python

Dashboards

Monitoring

SLOs

LLM eval

Agile teams

Tools

OpenTelemetry

Observability tools

Job description

NTT DATA in Charlotte, NC seeks an Observability & Evaluation Engineer to build telemetry, dashboards, and evaluation suites for Tachyon agent releases. You will ensure AI systems are monitored, evaluated, and supported with clear operational visibility.

The role emphasizes developing runbooks, alerts, SLOs, and readiness evidence, collaborating with platform engineers and product owners, and automating evidence collection for release readiness and governance.