Dallas-based Senior AI Observability & Evaluation Engineer

Quantum Technologies LLC

Dallas (TX)

On-site

USD 91,000 - 124,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Quantum Technologies LLC in Dallas, TX is seeking a Senior AI Observability & Evaluation Engineer to lead the design of observability for production AI systems, focusing on Generative AI, LLM monitoring, and RAG pipelines. The role requires deep Python experience, cloud familiarity (AWS/Azure/GCP), and strong knowledge of OpenTelemetry, Prometheus, Grafana, Docker, and Kubernetes.

The engineer will collaborate with AI teams to ensure reliability, cost control, and measurable AI quality across

Qualifications

  • 6+ years of experience in software engineering, Python, or AI/ML engineering.
  • Strong hands-on experience with Python and REST APIs.
  • Experience working with LLMs and Generative AI applications.
  • Strong knowledge of OpenTelemetry and distributed tracing.
  • Experience with LangSmith, Langfuse, or MLflow.
  • Knowledge of RAG pipelines, embeddings, and vector databases.
  • Experience with AI evaluation and performance monitoring.
  • Knowledge of Prometheus, Grafana, and monitoring tools.
  • Experience with cloud platforms such as AWS, Azure, or GCP.
  • Strong understanding of API monitoring, logging, and debugging.
  • Experience with Docker, Kubernetes, and CI/CD pipelines.

Responsibilities

  • Design and implement observability solutions for Generative AI applications.
  • Monitor LLM performance, latency, token usage, and application costs.
  • Develop evaluation frameworks for RAG and agentic AI applications.
  • Track hallucinations, response quality, and model reliability.Implement distributed tracing using OpenTelemetry.
  • Build dashboards and monitoring solutions using Grafana and Prometheus.
  • Troubleshoot production AI issues and improve application reliability.
  • Collaborate with AI engineers, platform teams, and architects.
  • Establish evaluation metrics and quality standards for enterprise AI applications.

Skills

Python
REST APIs
LLMs / Generative AI
OpenTelemetry
Distributed tracing
RAG pipelines / embeddings
AI evaluation / monitoring
Prometheus / Grafana
AWS / Azure / GCP
Docker / Kubernetes / CI/CD
Agent tracing

Tools

LangSmith
Langfuse
MLflow
OpenTelemetry
Prometheus
Grafana
Docker
Kubernetes
CI/CD

Job description

Quantum Technologies LLC in Dallas, TX is seeking a Senior AI Observability & Evaluation Engineer to lead the design of observability for production AI systems, focusing on Generative AI, LLM monitoring, and RAG pipelines. The role requires deep Python experience, cloud familiarity (AWS/Azure/GCP), and strong knowledge of OpenTelemetry, Prometheus, Grafana, Docker, and Kubernetes.

The engineer will collaborate with AI teams to ensure reliability, cost control, and measurable AI quality across

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Observability Engineer – LLM & RAG Monitoring
Senior AI Observability Engineer – LLM & RAG Monitoring

Robotics Technologies • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer (Production)
Senior AI Observability & Evaluation Engineer (Production)

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 65,000 - 107,000
Senior GenAI Observability & Evaluation Engineer
Senior GenAI Observability & Evaluation Engineer

MACHINE LEARNING TECHNOLOGIES LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior AI Observability Engineer: LLM Evaluation
Senior AI Observability Engineer: LLM Evaluation

CloudBigD • Dallas (TX)

On-site
USD 89,000 - 126,000
AI Observability & Evaluation Lead for Production ML
AI Observability & Evaluation Lead for Production ML

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability Engineer - LLM Monitoring
Senior AI Observability Engineer - LLM Monitoring

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

MACHINE LEARNING TECHNOLOGIES LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior Java AI Engineer & Architect
Senior Java AI Engineer & Architect

Quantum Technologies LLC • Atlanta (GA)

On-site
USD 111,000 - 159,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 65,000 - 107,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000