Production AI Reliability Engineer

KaiCalls; United States; Hybrid; USD 160,000 - 220,000; Full time

Job summary

KaiCalls is seeking an experienced AI reliability engineer to own the agent orchestration and evaluation infrastructure. You will build tracing, replay, datasets, and failure analysis to ensure trustworthy production AI behavior.

In this remote US role, you will design fallbacks, human-review boundaries, and manage releases, partnering with voice quality and telephony engineering. The position emphasizes security, reliability, and rigorous evaluation methods to reduce manual recovery.

Qualifications

• Production LLM or agent-system experience required.
• Strong software and reliability engineering background.
• Experience in evaluation design and statistical judgment.
• Security and privacy awareness is essential.

Responsibilities

• Own agent orchestration and evaluation infrastructure.
• Build tracing, replay, datasets, and failure analysis.
• Design fallbacks and human-review boundaries.
• Control model, prompt, and tool releases.

Skills

Production LLM experience

Reliability engineering

Evaluation design

Security & privacy

Job description

KaiCalls is seeking an experienced AI reliability engineer to own the agent orchestration and evaluation infrastructure. You will build tracing, replay, datasets, and failure analysis to ensure trustworthy production AI behavior.

In this remote US role, you will design fallbacks, human-review boundaries, and manage releases, partnering with voice quality and telephony engineering. The position emphasizes security, reliability, and rigorous evaluation methods to reduce manual recovery.