Engineering Manager – Evaluation & Observability

AgentsFlow; Northern (KY); Hybrid; USD 170,000 - 190,000; Full time

Job summary

AgentsFlow is seeking an Engineering Manager to lead the Evaluation & Observability team within our AI governance platform. You will shape how we measure model performance, safety, and reliability across enterprise AI deployments.

You will build and run evaluation and observability systems, mentor engineers (backend, data, ML), and ensure alignment with NIST-like governance standards. Remote US-based role with strong leadership impact.

Qualifications

• 8–12 years engineering experience.
• 3+ years managing teams.
• Familiarity with LLM/agent evaluation techniques.

Responsibilities

• Lead the evaluation, monitoring, and observability layer.
• Build model evaluation systems (accuracy, bias, drift, hallucination detection).
• Develop agent monitoring (inputs/outputs, decisions, tool usage).
• Create policy enforcement + audit logs.
• Design observability pipelines (real-time + batch).
• Lead a team of engineers (back-end, data, ML).

Skills

LLM evaluation

Observability platforms

Team leadership

Backend engineering

ML systems design

Job description

Engineering Manager – Evaluation & Observability

Lead the team building AI evaluation and observability systems that detect drift, policy violations and prompt injection across enterprise AI.

AI Governance Platform – Evaluation & Observability

Location

US / Remote

Experience

8-10 Years

USD 170,000 – 190,000

Posted 25 March 2026

Apply by 31 July 2026

About the Role

You will build and lead the evaluation, monitoring, and observability layer of our AI governance platform — core to trust, compliance, and auditability.

What You'll Do

• Build systems for model evaluation (accuracy, bias, drift, hallucination detection)
• Build agent monitoring (inputs/outputs, decisions, tool usage)
• Build policy enforcement + audit logs
• Design observability pipelines (real-time + batch)
• Define evaluation frameworks aligned with NIST AI Risk Management Framework
• Work on explainability, traceability, and audit artifacts
• Lead a team of engineers (back-end + data + ML)

What We’re Looking For

• 8–12 years engineering experience, 3+ years managing teams
• Familiarity with LLM/agent evaluation techniques
• Experience building observability platforms

Nice to Have

• Experience with AI governance / compliance systems
• Knowledge of EU AI Act technical requirements