AI / Agent Reliability Engineer

KaiCalls; United States; Hybrid; USD 160,000 - 220,000; Full time

Job summary

KaiCalls is seeking an experienced AI reliability engineer to own the agent orchestration and evaluation infrastructure. You will build tracing, replay, datasets, and failure analysis to ensure trustworthy production AI behavior.

In this remote US role, you will design fallbacks, human-review boundaries, and manage releases, partnering with voice quality and telephony engineering. The position emphasizes security, reliability, and rigorous evaluation methods to reduce manual recovery.

Qualifications

• Production LLM or agent-system experience required.
• Strong software and reliability engineering background.
• Experience in evaluation design and statistical judgment.
• Security and privacy awareness is essential.

Responsibilities

• Own agent orchestration and evaluation infrastructure.
• Build tracing, replay, datasets, and failure analysis.
• Design fallbacks and human-review boundaries.
• Control model, prompt, and tool releases.

Skills

Production LLM experience

Reliability engineering

Evaluation design

Security & privacy

Job description

The mission

Build the evaluation, tracing, fallback, and release systems that keep production AI behavior trustworthy.

What you will own

• Own agent orchestration and evaluation infrastructure.
• Build tracing, replay, datasets, and failure analysis.
• Design fallbacks and human-review boundaries.
• Control model, prompt, and tool releases.
• Partner with voice quality and telephony engineering.

What success looks like

• Versioned evaluations connect model behavior to customer outcomes.
• Manual recovery declines and failure categories are clear.
• AI changes have release gates and rollback.
• Production decisions are traceable without leaking sensitive data.

What we are looking for

• Production LLM or agent-system experience.
• Strong software and reliability engineering.
• Evaluation design and statistical judgment.
• Security and privacy awareness.

Helpful, not mandatory

• Realtime voice AI.
• Tool-calling systems.
• Python and TypeScript.
• Model routing and cost optimization.

Compensation

The base salary range is $160,000-$220,000 USD for a US-remote employee. Final compensation will reflect role level, location policy, experience, benefits, variable compensation where applicable, and equity. Contractor arrangements are scoped separately.

How we evaluate

We use the same core evidence for every candidate: comparable outcomes, ownership, customer judgment, role skill, communication, and learning velocity. The process normally includes a short screen, a paid and bounded work sample, structured interviews, and reference checks.

Example paid work sample: Design the detection, prevention, release, and rollback plan for a regression where the agent captures a caller's name but loses the service intent.

Your first 90 days

1. First 30 days: Inventory production failure modes and evaluations.
2. By 60 days: Establish replay and release discipline.
3. By 90 days: Materially reduce the leading manual-recovery category.