Monitoring And Evaluation Specialist

KTek Resourcing LLC; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

KTek Resourcing LLC is seeking a GenAI / LLM Observability & Evaluation Engineer to ensure reliability and performance of production AI agents and LLM-powered apps. You will implement evaluation frameworks, tracing, telemetry, and dashboards to monitor production systems, driving incident readiness and operational excellence.

The role requires hands-on experience with LLMs, evaluation tools, Python automation, and strong observability skills to optimize performance and reliability in a

Qualifications

• Hands-on work with LLMs, Generative AI, or Agentic AI.
• Experience evaluating LLMs, testing agents, and analyzing performance.
• Strong Python and test automation skills.
• Experience with observability, tracing, telemetry and monitoring.
• Ability to build dashboards and analyze application performance.
• Knowledge of SLOs, production operations, incident response, and reliability.

Responsibilities

• Implement LLM evaluation frameworks and agent testing workflows.
• Design and operate observability, tracing, and telemetry solutions.
• Build and maintain dashboards to monitor production AI agents.
• Collaborate with teams to improve reliability and release quality.

Skills

LLMs / Generative AI applications

LLM evaluation & agent testing

Python programming & test automation

Observability, tracing & telemetry

Metrics dashboards & performance分析

SLOs, production ops & incident readin

Tools

OpenTelemetry

Prometheus

Grafana

Datadog

Splunk

LangSmith

Langfuse

Arize Phoenix

LangChain

Google ADK

Job description

We are seeking a skilled GenAI / LLM Observability & Evaluation Engineer to help ensure the reliability, performance, and quality of production AI agents and LLM-powered applications.

The ideal candidate will have hands-on experience implementing LLM evaluation frameworks, distributed tracing, telemetry, monitoring dashboards, automated testing, and production reliability practices.

Required Skills

• Hands-on experience with LLMs, Generative AI, or Agentic AI applications.


• Experience with LLM evaluation, agent testing, and performance analysis.


• Strong Python programming and test automation skills.


• Experience implementing observability, tracing, telemetry, and monitoring solutions.


• Experience building metrics dashboards and analyzing application performance.


• Understanding of SLOs, production operations, incident response, and operational readiness.

Preferred Skills

• LLM evaluation tools such as DeepEval, Ragas, Promptfoo, or equivalent.


• AI observability platforms such as LangSmith, Langfuse, Arize Phoenix, or equivalent.


• OpenTelemetry, OpenInference, Prometheus, Grafana, Datadog, or Splunk.


• Experience with LangChain, LangGraph, Google ADK, or similar agent frameworks.


• Familiarity with CI/CD pipelines, automated regression testing, and release quality gates.