AI Observability & Evaluation Engineer

Mphasis; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Mphasis is seeking an Observability and Evaluation Engineer in Charlotte, NC to design and implement observability pipelines for AI/LLM workflows, and build robust evaluation and testing frameworks to ensure model quality and reliability.

You will instrument applications, develop offline/online evaluation suites, and work on performance and cost optimization across distributed microservices and AI orchestration layers.

Qualifications

• Strong proficiency in Python or Go for building custom evaluation scripts and telemetry hooks.
• Experience with AI observability platforms and telemetry stacks for tracking LLM/agent workflows.
• Knowledge of adversarial prompts and testing for LLM/system validation.

Responsibilities

• Build Observability Pipelines: instrument applications and LLM/agent pipelines using telemetry tools to capture traces, logs, and latency metrics.
• Design Evaluation Frameworks: create offline and online evaluation suites to benchmark accuracy, groundedness, toxicity, tool-use correctness, and reasoning-chain validity.
• Implement Regression & Drift Testing: build automated test harnesses and continuous evaluation gates to detect model degradation, data drift, or output anomalies before releases reach production.
• Root-Cause Analysis: investigate execution traces and multi-turn interaction failures to diagnose erratic system behaviors, API misparameters, or bottlenecks.
• Optimize Performance & Cost: monitor and balance operational telemetry relating to token consumption, execution speed, and infrastructure costs.

Skills

Python

Go

Testing methodologies

System design

Tools

OpenTelemetry

Prometheus

Grafana

Arize Phoenix

LangChain/LangSmith

Galileo

Braintrust

Job description

Mphasis is seeking an Observability and Evaluation Engineer in Charlotte, NC to design and implement observability pipelines for AI/LLM workflows, and build robust evaluation and testing frameworks to ensure model quality and reliability.

You will instrument applications, develop offline/online evaluation suites, and work on performance and cost optimization across distributed microservices and AI orchestration layers.