Senior AI Evaluation Engineer: Pipelines & Metrics

logicmonitor; San Francisco (CA); On-site; USD 150,000 - 190,000; Full time

Job summary

LogicMonitor is seeking a Senior AI Engineer, Evaluations, to design and build evaluation systems for Edwin AI. You will create production-grade evaluation pipelines, golden datasets, automated graders, and regression frameworks for AI agents, retrieval systems, tool integrations, and complex investigation workflows.

You will define quality metrics, build offline and online evaluation pipelines in Python, integrate with CI/CD, and mentor others while monitoring AI quality and behavior drift in

Qualifications

• 5+ years of experience in software engineering, ML, or a related field.

Responsibilities

• Define quality metrics for incident diagnostics, root-cause analysis, alert correlation, grounding, tool use, safety, and operational usefulness.

Skills

Python

AI evaluation

Experimentation

Quality frameworks

LLM evaluation

Tools

LangSmith

Arize Phoenix

Braintrust

DeepEval

Ragas

TruLens

OpenAI Evals

MLflow

Job description

LogicMonitor is seeking a Senior AI Engineer, Evaluations, to design and build evaluation systems for Edwin AI. You will create production-grade evaluation pipelines, golden datasets, automated graders, and regression frameworks for AI agents, retrieval systems, tool integrations, and complex investigation workflows.

You will define quality metrics, build offline and online evaluation pipelines in Python, integrate with CI/CD, and mentor others while monitoring AI quality and behavior drift in