Observability & Evaluation Engineer

NTT DATA Americas, Inc.; Charlotte (NC); On-site; USD 140,000 - 190,000; Full time

Job summary

NTT DATA in Charlotte, NC is seeking an Observability & Evaluation Engineer to build telemetry, tracing, dashboards, and evaluation suites for Tachyon agent releases, ensuring production AI systems are monitored and supported with clear operational visibility.

You will implement observability for LLM-powered applications, develop dashboards and alerts, define monitoring requirements with Product Owners, and automate evidence collection for release readiness.

Qualifications

• 7+ years of engineering experience in observability, monitoring, test automation, platform operations, or AI/ML systems.
• 5+ years of strong hands-on Python experience.
• 5+ years of experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.
• 5+ years of understanding LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.
• 5+ years of experience working in Agile engineering teams and production support environments.

Responsibilities

• Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
• Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
• Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
• Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
• Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
• Create runbooks and support documentation for priority agent releases.
• Analyze production behavior and recommend improvements to reliability, performance, and quality.

Skills

Python

Observability

Dashboards/Monitoring

LLM evaluation

Agile teams

Tools

Prometheus

OpenTelemetry

Kubernetes

CI/CD

Job description

Company Overview: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization.

We are currently seeking a Observability & Evaluation Engineer to join our team in Charlotte, North Carolina (US-NC), United States (US).

Job Description:

Observability & Evaluation Engineer will build telemetry, tracing, dashboards, evaluation suites, alerts, service objectives, runbooks, and readiness evidence for Tachyon agent releases. This role ensures production AI systems can be monitored, evaluated, improved, and supported with clear operational visibility.

Key Responsibilities

• Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
• Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
• Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
• Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
• Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
• Create runbooks and support documentation for priority agent releases.
• Analyze production behavior and recommend improvements to reliability, performance, and quality.

Required Qualifications

• 7+ years of engineering experience with observability, monitoring, test automation, platform operations, or AI/ML systems.
• 5+ years of strong hands‑on Python experience.
• 5+ years of Experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.
• 5+ years of Understanding of LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.
• 5+ years of Experience working in Agile engineering teams and production support environments.

Required Skills / Knowledge

• Python, telemetry, tracing, monitoring, dashboards, alerting, SLOs, evaluation frameworks, test automation, and production operations.
• Understanding of LLMs, agents, RAG, prompt performance, retrieval quality, latency, and reliability metrics.
• Experience with observability tools and open telemetry concepts.

Preferred Qualifications

• Experience with GenAI observability, AI evaluation tools, ML monitoring, or platform reliability engineering.
• Experience in regulated environments with evidence and readiness documentation.
• Kubernetes, cloud platforms, and CI/CD experience.

Expected Outcomes

• Operational dashboards and evaluation suites for priority agent releases.
• Clear readiness evidence, alerts, SLOs, and runbooks.
• Improved quality, reliability, and trust in production Agentic AI systems.

About NTT DATA:

NTT DATA is a $30 billion trusted global innovator of business and technology services. We serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize and transform for long term success. As a Global Top Employer, we have diverse experts in more than 50 countries and a robust partner ecosystem of established and start‑up companies. Our services include business and technology consulting, data and artificial intelligence, industry solutions, as well as the development, implementation and management of applications, infrastructure and connectivity. We are one of the leading providers of digital and AI infrastructure in the world. NTT DATA is a part of NTT Group, which invests over $3.6 billion each year in R&D to help organizations and society move confidently and sustainably into the digital future. Visit us at us.nttdata.com

NTT DATA endeavors to make https://us.nttdata.com accessible to any and all users. If you would like to contact us regarding the accessibility of our website or need assistance completing the application process, please contact us at https://us.nttdata.com/en/contact-us. This contact information is for accommodation requests only and cannot be used to inquire about the status of applications. NTT DATA is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status. For our EEO Policy Statement, please click here. If youd like more information on your EEO rights under the law, please click here. For Pay Transparency information, please click here.