Senior AI Engineer, Agent Evaluation and Governance

Findjobit; United States; Remote; USD 123,000 - 167,000; Full time

Job summary

Niuro is hiring a Senior AI Engineer for an enterprise AI platform company where AI agents take real actions: sending messages, updating records and preparing decisions. You will own the evaluation and governance layer that decides when each agent capability is ready to go live and keeps it safe in production.

You will design evaluation sets, thresholds, and go-live gates; detect regressions; build safeguards; log every model call; and translate governance rules into executable validation with

Qualifications

• 5+ years of software engineering experience.
• Production experience with tool-using, multi-step LLM agents.
• Evaluation sets, quality thresholds and regression detection in production.
• Applied statistics: review sample sizing, confidence bounds, error-rate estimation.
• Strong Python or TypeScript, with testing and code review.
• C1 English.

Responsibilities

• Design evaluation sets, metrics, quality thresholds and go-live gates per agent capability.
• Detect regressions after model, prompt or instruction changes.
• Build runtime safeguards in code: approvals, data restrictions, deterministic checks, kill switches and quality-based suspension.
• Log and trace every model call for accountability and auditability.
• Turn governance rules and acceptance scenarios into executable validation with platform, cloud and product teams.
• Write clear design notes before implementation.

Skills

5+ years of software engineering

Production experience with multi-step

Evaluation sets and regression

Applied statistics

Python or TypeScript

C1 English

Tools

Python

TypeScript

Job description

Niuro is hiring a Senior AI Engineer for an enterprise AI platform company where AI agents take real actions: sending messages, updating records and preparing decisions. You will own the evaluation and governance layer that decides when each agent capability is ready to go live and keeps it safe in production.

You will

• Design evaluation sets, metrics, quality thresholds and go-live gates per agent capability.

• Detect regressions after model, prompt or instruction changes.

• Build runtime safeguards in code: approvals, data restrictions, deterministic checks, kill switches and quality-based suspension.

• Log and trace every model call for accountability and auditability.

• Turn governance rules and acceptance scenarios into executable validation with platform, cloud and product teams.

• Write clear design notes before implementation.

You bring

• 5+ years of software engineering experience.

• Production experience with tool-using, multi-step LLM agents.

• Evaluation sets, quality thresholds and regression detection in production.

• Applied statistics: review sample sizing, confidence bounds, error-rate estimation.

• Strong Python or TypeScript, with testing and code review.

• C1 English.

Nice to have: knowledge graphs and ontologies (RDF, OWL, SHACL, LinkML), RAG, human-in-the-loop systems, access control, multi-provider LLM integration, financial services.

Contract: full-time, fully remote, monthly contract in USD.