Observability and Evaluation Engineer

Mphasis; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Mphasis is seeking an Observability and Evaluation Engineer in Charlotte, NC to design and implement observability pipelines for AI/LLM workflows, and build robust evaluation and testing frameworks to ensure model quality and reliability.

You will instrument applications, develop offline/online evaluation suites, and work on performance and cost optimization across distributed microservices and AI orchestration layers.

Qualifications

• Strong proficiency in Python or Go for building custom evaluation scripts and telemetry hooks.
• Experience with AI observability platforms and telemetry stacks for tracking LLM/agent workflows.
• Knowledge of adversarial prompts and testing for LLM/system validation.

Responsibilities

• Build Observability Pipelines: instrument applications and LLM/agent pipelines using telemetry tools to capture traces, logs, and latency metrics.
• Design Evaluation Frameworks: create offline and online evaluation suites to benchmark accuracy, groundedness, toxicity, tool-use correctness, and reasoning-chain validity.
• Implement Regression & Drift Testing: build automated test harnesses and continuous evaluation gates to detect model degradation, data drift, or output anomalies before releases reach production.
• Root-Cause Analysis: investigate execution traces and multi-turn interaction failures to diagnose erratic system behaviors, API misparameters, or bottlenecks.
• Optimize Performance & Cost: monitor and balance operational telemetry relating to token consumption, execution speed, and infrastructure costs.

Skills

Python

Go

Testing methodologies

System design

Tools

OpenTelemetry

Prometheus

Grafana

Arize Phoenix

LangChain/LangSmith

Galileo

Braintrust

Job description

Core Responsibilities

• Build Observability Pipelines: Instrument applications and LLM/agent pipelines using telemetry tools (like OpenTelemetry, Arize, Galileo, or LangSmith) to capture execution traces, run logs, and latency metrics. [1]
• Design Evaluation Frameworks: Create offline and online evaluation suites to benchmark accuracy, groundedness, toxicity, tool-use correctness, and reasoning-chain validity. []
• Implement Regression & Drift Testing: Build automated test harnesses and continuous evaluation gates to detect model degradation, data drift, or output anomalies before releases reach production. [1]
• Root-Cause Analysis: Investigate execution traces and multi-turn interaction failures to diagnose erratic system behaviors, API misparameters, or bottlenecks. []
• Optimize Performance & Cost: Monitor and balance operational telemetry relating to token consumption, execution speed, and infrastructure costs. [1]

Key Skills & Requirements

• Programming: Strong proficiency in Python or Go for building custom evaluation scripts and hooking into telemetry SDKs.
• Tools & Stacks: Experience with OpenTelemetry, Prometheus, Grafana, or dedicated AI observability platforms (e.g., Arize Phoenix, LangChain/LangSmith, Galileo, Braintrust).
• Testing Methodologies: Designing adversarial prompts, red-teaming protocols, and golden datasets for LLM/system validation.
• System Design: Understanding of distributed microservices or LLM orchestration layers (LangChain, LlamaIndex, AutoGen). [1, 2, 3]

An Observability and Evaluation Engineer (often specialized in AI/LLM systems) designs the tracking, tracing, and testing frameworks that monitor production behavior, measure output quality, and catch regressions in complex software or agentic AI workflows. [1, 2, 3, 4]

If you are tailoring this for a specific application, would you like me to focus this job description more heavily on traditional cloud/microservices observability or Generative AI / LLM agent evaluation?