Voice AI Quality & Evaluation Analyst

KaiCalls; Northern (KY); Hybrid; USD 70,000 - 100,000; Full time

Job summary

KaiCalls is seeking a meticulous QA reviewer to sample and review real calls under privacy rules and to maintain rubrics for accuracy, intent capture, latency, tone, transfer, consent and outcome.

You will distinguish genuine business intent from demos, report failure clusters, and partner with engineering on remediation. This remote role supports evaluating calls across verticals and languages with emphasis on evidence-backed writing and handling sensitive data.

Qualifications

• Exceptional listening and analytical judgment.
• Experience in QA, conversation review, trust and safety, or applied AI evaluation.
• Evidence-backed writing and careful handling of sensitive data.
• Comfort working with imperfect data.

Responsibilities

• Sample and review real calls under privacy rules.
• Maintain rubrics for accuracy, intent capture, latency, tone, transfer, consent, and outcome.
• Distinguish genuine business intent from demos and test traffic.
• Report failure clusters and partner with engineering on remediation.
• Report evaluation findings to drive measurable product improvements.

Skills

Active listening

Analytical judgment

QA/review experience

Data handling

Job description

The mission

Build the evidence loop that shows how Kai performs on real calls and what must improve next.

What you will own

• Sample and review real calls under privacy rules.
• Maintain rubrics for accuracy, intent capture, latency, tone, transfer, consent, and outcome.
• Distinguish genuine business intent from demos and test traffic.
• Report failure clusters and partner with engineering on remediation.

What success looks like

• Representative weekly evaluation sets span verticals and languages.
• Reviewers calibrate consistently on the same calls.
• Serious complaints receive source, transcript, audio, and metadata review.
• Evaluation findings produce measurable product improvements.

What we are looking for

• Exceptional listening and analytical judgment.
• Experience in QA, conversation review, trust and safety, or applied AI evaluation.
• Evidence-backed writing and careful handling of sensitive data.
• Comfort working with imperfect data.

Helpful, not mandatory

• Professional Spanish.
• Contact-center QA.
• Speech or LLM evaluation.
• Basic data analysis.

Compensation

The base salary range is $70,000-$100,000 USD for a US-remote employee. Final compensation will reflect role level, location policy, experience, benefits, variable compensation where applicable, and equity. Contractor arrangements are scoped separately.

How we evaluate

We use the same core evidence for every candidate: comparable outcomes, ownership, customer judgment, role skill, communication, and learning velocity. The process normally includes a short screen, a paid and bounded work sample, structured interviews, and reference checks.

Example paid work sample: Score five redacted calls, identify rubric weaknesses, separate real intent from test traffic, and recommend one product change.

Your first 90 days

1. First 30 days: Calibrate against known good and bad calls.
2. By 60 days: Run the weekly evaluation program.
3. By 90 days: Demonstrate one meaningful quality improvement tied to the evidence.