AI Observability Architect & Evaluation Engineer

Mphasis; Charlotte (NC); On-site; USD 120,000 - 160,000; Full time

Job summary

Mphasis in Charlotte, NC is seeking an experienced Observability and Evaluation Engineer to design, implement, and optimize observability, monitoring, and evaluation frameworks for AI/ML, GenAI, and enterprise applications.

The ideal candidate will build end-to-end telemetry, monitoring, tracing, and evaluation solutions that ensure reliability, performance, scalability, and quality of AI-driven systems, including LLMs, RAG applications, and Agentic AI platforms.

Qualifications

• Bachelor's or master's degree in computer science, engineering, data science, or related field.
• 6+ years of experience in Observability, Site Reliability Engineering, Platform Engineering, or AI/ML Engineering.
• 2+ years of experience supporting GenAI/LLM applications.
• Strong experience with telemetry, monitoring, tracing, and evaluation frameworks.
• Experience with cloud-native technologies and Kubernetes.
• Strong analytical and troubleshooting skills.

Responsibilities

• Design and implement observability frameworks for AI/ML and enterprise applications.
• Develop monitoring, telemetry, logging, and distributed tracing solutions.
• Build evaluation pipelines to measure accuracy, relevance, latency, hallucination, and overall quality of AI models and applications.
• Define and track SLIs, SLOs, and KPIs for production systems.
• Implement automated model evaluation and benchmarking frameworks.
• Monitor and troubleshoot production issues across cloud and Kubernetes environments.
• Analyze system performance and recommend optimization strategies.
• Develop dashboards and real-time monitoring solutions using industry‑standard tools.
• Collaborate with Data Scientists, ML Engineers, Platform Engineers, and Product Teams.
• Create alerting mechanisms and incident response procedures.
• Support GenAI, RAG, and Agentic AI applications through continuous evaluation and monitoring.

Skills

Observability

Site Reliability Engineering

Platform Engineering

AI/ML Engineering

Telemetry/Monitoring/Tracing

Cloud-native technologies

Kubernetes

Education

Bachelor's or master's degree in Computer Science/Engineering/Data Science or related field

Tools

Kubernetes

Cloud-native tech

Job description

Mphasis in Charlotte, NC is seeking an experienced Observability and Evaluation Engineer to design, implement, and optimize observability, monitoring, and evaluation frameworks for AI/ML, GenAI, and enterprise applications.

The ideal candidate will build end-to-end telemetry, monitoring, tracing, and evaluation solutions that ensure reliability, performance, scalability, and quality of AI-driven systems, including LLMs, RAG applications, and Agentic AI platforms.