Observability and Evaluation Engineer

Mphasis; Charlotte (NC); On-site; USD 120,000 - 160,000; Full time

Job summary

Mphasis in Charlotte, NC is seeking an experienced Observability and Evaluation Engineer to design, implement, and optimize observability, monitoring, and evaluation frameworks for AI/ML, GenAI, and enterprise applications.

The ideal candidate will build end-to-end telemetry, monitoring, tracing, and evaluation solutions that ensure reliability, performance, scalability, and quality of AI-driven systems, including LLMs, RAG applications, and Agentic AI platforms.

Qualifications

• Bachelor's or master's degree in computer science, engineering, data science, or related field.
• 6+ years of experience in Observability, Site Reliability Engineering, Platform Engineering, or AI/ML Engineering.
• 2+ years of experience supporting GenAI/LLM applications.
• Strong experience with telemetry, monitoring, tracing, and evaluation frameworks.
• Experience with cloud-native technologies and Kubernetes.
• Strong analytical and troubleshooting skills.

Responsibilities

• Design and implement observability frameworks for AI/ML and enterprise applications.
• Develop monitoring, telemetry, logging, and distributed tracing solutions.
• Build evaluation pipelines to measure accuracy, relevance, latency, hallucination, and overall quality of AI models and applications.
• Define and track SLIs, SLOs, and KPIs for production systems.
• Implement automated model evaluation and benchmarking frameworks.
• Monitor and troubleshoot production issues across cloud and Kubernetes environments.
• Analyze system performance and recommend optimization strategies.
• Develop dashboards and real-time monitoring solutions using industry‑standard tools.
• Collaborate with Data Scientists, ML Engineers, Platform Engineers, and Product Teams.
• Create alerting mechanisms and incident response procedures.
• Support GenAI, RAG, and Agentic AI applications through continuous evaluation and monitoring.

Skills

Observability

Site Reliability Engineering

Platform Engineering

AI/ML Engineering

Telemetry/Monitoring/Tracing

Cloud-native technologies

Kubernetes

Education

Bachelor's or master's degree in Computer Science/Engineering/Data Science or related field

Tools

Kubernetes

Cloud-native tech

Job description

Job Summary We are seeking an experienced Observability and Evaluation Engineer to design, implement, and optimize observability, monitoring, and evaluation frameworks for AI/ML, GenAI, and enterprise applications. The ideal candidate will build end-to-end telemetry, monitoring, tracing, and evaluation solutions that ensure reliability, performance, scalability, and quality of AI-driven systems, including LLMs, RAG applications, and Agentic AI platforms.

About the Role

We are seeking an experienced Observability and Evaluation Engineer to design, implement, and optimize observability, monitoring, and evaluation frameworks for AI/ML, GenAI, and enterprise applications. The ideal candidate will build end-to-end telemetry, monitoring, tracing, and evaluation solutions that ensure reliability, performance, scalability, and quality of AI-driven systems, including LLMs, RAG applications, and Agentic AI platforms.

Responsibilities

• Design and implement observability frameworks for AI/ML and enterprise applications.
• Develop monitoring, telemetry, logging, and distributed tracing solutions.
• Build evaluation pipelines to measure accuracy, relevance, latency, hallucination, and overall quality of AI models and applications.
• Define and track SLIs, SLOs, and KPIs for production systems.
• Implement automated model evaluation and benchmarking frameworks.
• Monitor and troubleshoot production issues across cloud and Kubernetes environments.
• Analyze system performance and recommend optimization strategies.
• Develop dashboards and real-time monitoring solutions using industry‑standard tools.
• Collaborate with Data Scientists, ML Engineers, Platform Engineers, and Product Teams.
• Create alerting mechanisms and incident response procedures.
• Support GenAI, RAG, and Agentic AI applications through continuous evaluation and monitoring.

Qualifications

• Bachelor's or master's degree in computer science, Engineering, Data Science, or related field.
• 6+ years of experience in Observability, Site Reliability Engineering, Platform Engineering, or AI/ML Engineering.
• 2+ years of experience supporting GenAI/LLM applications.
• Strong experience with telemetry, monitoring, tracing, and evaluation frameworks.
• Experience with cloud‑native technologies and Kubernetes platforms.
• Strong analytical and troubleshooting skills.