AI Evaluation & Observability Lead (Remote)

AgentsFlow; Northern (KY); On-site; USD 170,000 - 190,000; Full time

Job summary

AgentsFlow is seeking an Engineering Manager to lead the Evaluation & Observability team within our AI governance platform. You will shape how we measure model performance, safety, and reliability across enterprise AI deployments.

You will build and run evaluation and observability systems, mentor engineers (backend, data, ML), and ensure alignment with NIST-like governance standards. Remote US-based role with strong leadership impact.

Qualifications

• 8–12 years engineering experience.
• 3+ years managing teams.
• Familiarity with LLM/agent evaluation techniques.

Responsibilities

• Lead the evaluation, monitoring, and observability layer.
• Build model evaluation systems (accuracy, bias, drift, hallucination detection).
• Develop agent monitoring (inputs/outputs, decisions, tool usage).
• Create policy enforcement + audit logs.
• Design observability pipelines (real-time + batch).
• Lead a team of engineers (back-end, data, ML).

Skills

LLM evaluation

Observability platforms

Team leadership

Backend engineering

ML systems design

Job description

AgentsFlow is seeking an Engineering Manager to lead the Evaluation & Observability team within our AI governance platform. You will shape how we measure model performance, safety, and reliability across enterprise AI deployments.

You will build and run evaluation and observability systems, mentor engineers (backend, data, ML), and ensure alignment with NIST-like governance standards. Remote US-based role with strong leadership impact.