Senior AI Observability & Evaluation Engineer (Production)

Ethereum Technologies LLC

Dallas (TX)

On-site

USD 65,000 - 107,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Ethereum Technologies is seeking a Senior AI Observability & Evaluation Engineer in Dallas, TX to design and implement observability for Generative AI applications. You will monitor LLM performance, latency, and costs while developing robust evaluation frameworks for RAG and agentic AI workflows.

The role demands hands-on Python, OpenTelemetry, and experience with Grafana/Prometheus, Docker, and Kubernetes.

Qualifications

  • 6+ years in software engineering, Python, or AI/ML engineering.
  • Strong hands-on Python and REST APIs experience.
  • Experience with LLMs and Generative AI applications.
  • Knowledge of OpenTelemetry and distributed tracing.
  • Experience with monitoring tools and CI/CD pipelines.
  • Cloud platforms experience (AWS/Azure/GCP).

Responsibilities

  • Design and implement observability solutions for Generative AI apps.
  • Monitor LLM performance, latency, token usage, and costs.
  • Develop evaluation frameworks for RAG and agentic AI.
  • Track hallucinations, response quality, and model reliability.
  • Implement distributed tracing with OpenTelemetry.
  • Build dashboards with Grafana and Prometheus.
  • Troubleshoot production AI issues and boost reliability.
  • Collaborate with AI engineers and platform teams.

Skills

Python
LLM monitoring
Observability concepts
Distributed tracing
AI evaluation
Production AI
REST APIs
MLOps
Cloud platforms
Docker
Kubernetes

Tools

OpenTelemetry
Prometheus
Grafana
LangSmith
Langfuse
MLflow

Job description

Ethereum Technologies is seeking a Senior AI Observability & Evaluation Engineer in Dallas, TX to design and implement observability for Generative AI applications. You will monitor LLM performance, latency, and costs while developing robust evaluation frameworks for RAG and agentic AI workflows.

The role demands hands-on Python, OpenTelemetry, and experience with Grafana/Prometheus, Docker, and Kubernetes.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior GenAI Observability & Evaluation Engineer
Senior GenAI Observability & Evaluation Engineer

MACHINE LEARNING TECHNOLOGIES LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior AI Observability Engineer – LLM & RAG Monitoring
Senior AI Observability Engineer – LLM & RAG Monitoring

Robotics Technologies • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability Engineer - LLM Monitoring
Senior AI Observability Engineer - LLM Monitoring

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability Engineer: LLM Evaluation
Senior AI Observability Engineer: LLM Evaluation

CloudBigD • Dallas (TX)

On-site
USD 89,000 - 126,000
AI Observability & Evaluation Lead for Production ML
AI Observability & Evaluation Lead for Production ML

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Dallas-based Senior AI Observability & Evaluation Engineer
Dallas-based Senior AI Observability & Evaluation Engineer

Quantum Technologies LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

MACHINE LEARNING TECHNOLOGIES LLC • Dallas (TX)

On-site
USD 91,000 - 124,000
Senior Gen AI Data Scientist - Production AI & Observability
Senior Gen AI Data Scientist - Production AI & Observability

Ethereum Technologies • Houston (TX)

On-site
USD 83,000 - 118,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

CloudBigD • Dallas (TX)

On-site
USD 89,000 - 126,000
Senior AI Observability & Evaluation Engineer
Senior AI Observability & Evaluation Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000