Senior AI Observability Engineer – LLM & RAG Monitoring

Robotics Technologies

Dallas (TX)

On-site

USD 89,000 - 126,000

Full time

35 hours ago
Be an early applicant

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Robotics Technologies LLC in Dallas, TX is seeking an experienced Senior AI Observability and Evaluation Engineer to design observability for Generative AI applications. You will monitor LLM performance, latency, token usage, and costs in production.

Requirements include strong Python, OpenTelemetry, Prometheus/Grafana, Docker and Kubernetes, plus cloud experience (AWS/Azure/GCP). You will build evaluation frameworks for RAG/agentic AI and collaborate with AI engineers to ensure reliable,

Qualifications

  • 6+ years in software/AI engineering or similar.
  • Proficient with Python and REST APIs.
  • Experience with LLMs and Generative AI applications.

Responsibilities

  • Design and implement observability solutions for Generative AI apps.
  • Monitor LLM performance, latency, token usage, and costs.
  • Develop evaluation frameworks for RAG and agentic AI.
  • Track hallucinations, response quality, and model reliability.
  • Implement distributed tracing using OpenTelemetry.
  • Build dashboards with Grafana and Prometheus.
  • Troubleshoot production AI issues and improve reliability.
  • Collaborate with AI engineers, platform teams, and architects.

Skills

Python
REST APIs
LLMs/Generative AI
OpenTelemetry
Distributed tracing
MLflow
LangSmith
Langfuse
Embeddings
Vector databases
Docker
Kubernetes
CI/CD
AWS
GCP
Azure
API monitoring
Prometheus
Grafana

Tools

OpenTelemetry
Prometheus
Grafana
Docker
Kubernetes
MLflow
LangSmith
Langfuse
Vector databases
Embeddings
AWS
Azure
GCP

Job description

Robotics Technologies LLC in Dallas, TX is seeking an experienced Senior AI Observability and Evaluation Engineer to design observability for Generative AI applications. You will monitor LLM performance, latency, token usage, and costs in production.

Requirements include strong Python, OpenTelemetry, Prometheus/Grafana, Docker and Kubernetes, plus cloud experience (AWS/Azure/GCP). You will build evaluation frameworks for RAG/agentic AI and collaborate with AI engineers to ensure reliable,

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Observability Engineer - LLM Monitoring
Senior AI Observability Engineer - LLM Monitoring

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability Engineer: LLM Evaluation
Senior AI Observability Engineer: LLM Evaluation

CloudBigD • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior GenAI Observability & Evaluation Engineer
Senior GenAI Observability & Evaluation Engineer

MACHINE LEARNING TECHNOLOGIES LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior AI Observability & Evaluation Engineer (Production)
Senior AI Observability & Evaluation Engineer (Production)

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 65,000 - 107,000
AI Observability & Evaluation Lead for Production ML
AI Observability & Evaluation Lead for Production ML

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Dallas-based Senior AI Observability & Evaluation Engineer
Dallas-based Senior AI Observability & Evaluation Engineer

Quantum Technologies LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

CloudBigD • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 65,000 - 107,000