Senior AI Observability & Evaluation Engineer

MACHINE LEARNING TECHNOLOGIES LLC

Dallas (TX)

On-site

USD 91,000 - 124,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

MACHINE LEARNING TECHNOLOGIES LLC in Dallas, TX seeks a Senior AI Observability & Evaluation Engineer to design and implement observability for Generative AI applications. You will monitor LLM performance, latency, token usage, and costs; develop evaluation frameworks for RAG and agentic AI; track hallucinations and model reliability, and implement distributed tracing with OpenTelemetry.

You will build dashboards with Grafana/Prometheus, troubleshoot production AI issues, and collaborate with AI

Qualifications

  • 6+ years of software engineering, Python, or AI/ML engineering.
  • Strong hands-on Python and REST APIs.
  • Experience with LLMs and Generative AI applications.
  • Strong knowledge of OpenTelemetry and distributed tracing.
  • Experience with LangSmith, Langfuse, or MLflow.
  • Knowledge of RAG pipelines, embeddings, and vector databases.
  • Experience with AI evaluation and performance monitoring.
  • Knowledge of Prometheus, Grafana, and monitoring tools.
  • Experience with AWS, Azure, or GCP.
  • Docker, Kubernetes, and CI/CD pipelines.
  • LLM observability and agent tracing.
  • RAG evaluation using RAGAS, DeepEval, or TruLens.
  • AI quality assessment and hallucination detection.
  • Prompt evaluation and version management.
  • Agentic AI workflow monitoring.
  • MCP instrumentation and monitoring.
  • Token usage, latency, and cost tracking.
  • AI guardrails and safety evaluation.
  • OpenTelemetry-based LLM tracing.
  • Production AI reliability and performance optimization.

Responsibilities

  • Design and implement observability solutions for Generative AI applications.
  • Monitor LLM performance, latency, token usage, and application costs.
  • Develop evaluation frameworks for RAG and agentic AI applications.
  • Track hallucinations, response quality, and model reliability.
  • Implement distributed tracing using OpenTelemetry.
  • Build dashboards and monitoring solutions using Grafana and Prometheus.
  • Troubleshoot production AI issues and improve application reliability.
  • Collaborate with AI engineers, platform teams, and architects.
  • Establish evaluation metrics and quality standards for enterprise AI applications.

Skills

Python
REST APIs
LLMs
OpenTelemetry
LangSmith
Langfuse
MLflow
RAG pipelines
Embeddings
Vector databases
AI evaluation
Prometheus
Grafana
AWS
Azure
GCP
Docker
Kubernetes
CI/CD
LLM observability
Agent tracing
RAGAS
DeepEval
TruLens
AI quality
Hallucination detection
Prompt evaluation
Version management
Agentic AI workflow
MCP instrumentation
Token usage
Latency
Cost tracking
AI guardrails
OpenTelemetry-based LLM tracing
Production AI reliability

Tools

OpenTelemetry
Prometheus
Grafana
LangSmith
Langfuse
MLflow
Docker
Kubernetes
CI/CD

Job description

Job Description

Job Overview

Job Title:Senior AI Observability & Evaluation Engineer

Location:Dallas, TX

Duration: 15 Months + Extension

Bill Rate: $78/hour (C2C)

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Required Skills
  • 6+ years of experience in software engineering, Python, or AI/ML engineering.
  • Strong hands-on experience with Python and REST APIs.
  • Experience working with LLMs and Generative AI applications.
  • Strong knowledge of OpenTelemetry and distributed tracing.
  • Experience with LangSmith, Langfuse, or MLflow.
  • Knowledge of RAG pipelines, embeddings, and vector databases.
  • Experience with AI evaluation and performance monitoring.
  • Knowledge of Prometheus, Grafana, and monitoring tools.
  • Experience with cloud platforms such as AWS, Azure, or GCP.
  • Strong understanding of API monitoring, logging, and debugging.
  • Experience with Docker, Kubernetes, and CI/CD pipelines.
  • Recent / Modern Skills (2025–2026)
  • LLM observability and agent tracing.
  • RAG evaluation using RAGAS, DeepEval, or TruLens.
  • AI quality assessment and hallucination detection.
  • Prompt evaluation and version management.
  • Agentic AI workflow monitoring.
  • MCP instrumentation and monitoring.
  • Token usage, latency, and cost tracking.
  • AI guardrails and safety evaluation.
  • OpenTelemetry-based LLM tracing.
  • Production AI reliability and performance optimization.
Responsibilities
  • Design and implement observability solutions for Generative AI applications.
  • Monitor LLM performance, latency, token usage, and application costs.
  • Develop evaluation frameworks for RAG and agentic AI applications.
  • Track hallucinations, response quality, and model reliability.
  • Implement distributed tracing using OpenTelemetry.
  • Build dashboards and monitoring solutions using Grafana and Prometheus.
  • Troubleshoot production AI issues and improve application reliability.
  • Collaborate with AI engineers, platform teams, and architects.
  • Establish evaluation metrics and quality standards for enterprise AI applications.
Equal Opportunity Employer

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

CloudBigD • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 65,000 - 107,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Quantum Technologies LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Robotics Technologies • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior GenAI Observability & Evaluation Engineer
Senior GenAI Observability & Evaluation Engineer

MACHINE LEARNING TECHNOLOGIES LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior AI Observability Engineer: LLM Evaluation
Senior AI Observability Engineer: LLM Evaluation

CloudBigD • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability Engineer - LLM Monitoring
Senior AI Observability Engineer - LLM Monitoring

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
AI Engineer
AI Engineer

Imigrus • Plano (TX)

On-site
USD 140,000 - 180,000