AI call quality analyst (manual) in health-tech

ТРТ; United States; Remote; USD 90,000 - 130,000; Full time

Benefits offered by this job

Unlimited PTO

Company hardware

Job summary

TRT is seeking an AI Call Quality Analyst to own the quality of AI-driven patient communications for US patients. You will define what good looks like, review AI voice/chat interactions, and drive fixes across agents, flows, and test suites.

Responsibilities include maintaining the quality rubric, monitoring production calls, and reporting trends for stakeholders. English at C1+ level and Russian fluency are required.

Qualifications

• 3+ years in QA / quality analysis of customer communications.
• Experience evaluating LLM/agent quality with human annotation.
• Able to read transcripts and JSON logs; basic data literacy.
• Hands-on prompt engineering for LLM agents.
• English C1+; Russian fluency.

Responsibilities

• Own the quality rubric for AI-driven patient communications and scoring.
• Review live interactions, drive fixes with the AI team, and ensure verification.
• Report trends, dashboards, and readiness to non-technical stakeholders.

Skills

QA methods

LLM evaluation

Prompt engineering

Data literacy

Communication skills

English (C1+)

Russian fluency

Tools

SQL

Job description

Описание

A stealth-mode AI-powered Cloud-Native Health-Tech company is looking for an AI Call Quality Analyst. The Business Domain revolves around providing Better Care for the US Patients; meaning, it’s about Treating Patients well. The company was founded more than five years ago. It’s not vaporware, their platform supports US Physician Networks (IPAs) by enabling Smarter, Risk-Adjusted, and more Predictive Care that improves real patient outcomes.

Задачи

Own the quality of our AI-driven patient communications - define what "good" means, catch where it breaks, and drive the fixes across the agent, the flow, and the checks. review AI voice/chat interactions and score them against a defined behavioural rubric. own the automated checks and the human review loop; keep the quality signal trustworthy. turn findings into concrete changes to the agent, the conversation flow, and the test suite. Define and maintain the quality rubric - the behavioural checks (off-script, hallucination, resolution, emergency handling, "cringe"/weird behaviour) and their pass/fail criteria. Monitor production calls - review a smart sample, run human annotation, and measure & correct the drift of the automated LLM judge. Root-cause and route every defect - decide whether it belongs to the agent prompt, the conversation flow, the data pipeline, or the telephony config. Drive fixes with the AI team - bring prioritized, reproducible findings to engineers, propose the prompt change where the defect sits in a prompt, and add regression/autotest cases before each release. Align changes with the business - agree flow and script changes with clinical/ops stakeholders, balancing patient experience, compliance, and outcomes. Report quality in plain language - trends, dashboards, and release-readiness calls for non-technical stakeholders.

Требования

3+ years in QA / quality analysis of customer communications (contact-center, CX, or conversational AI) - monitoring and scoring live interactions against rubrics, giving actionablefeedback, and owning it through to verified implementation (not just reporting – confirming the fix shipped and worked). Experience running an evaluation loop for LLM/agent quality - LLM-as-a-judge, human annotation, regression on golden sets. Comfortable reading transcripts and structured logs (JSON) and using basic SQL / data literacy to slice metrics. Hands-on prompt engineering - writing, versioning and iterating production prompts: system prompts for LLM voice/chat agents, and evaluator prompts for LLM-as-a-judge checks. Hands-on experience evaluating voice and/or chat AI agents (RetellAI or similar LLM-agent platforms). Practical understanding of conversation flows, intents, and failure modes - barge-in, ASR/TTS errors, voicemail handling, escalation and fallback paths. Strong analytical and communication skills - translate technical findings into business impact and back and own the quality bar across teams without formal authority (bridge business AI team). English - C1 or higher; ability to present work and lead discussions with US-based teammates, customers, and stakeholders. Fluent Russian for collaboration with Russian-speaking teammates. Будет плюсом: Experience in Healthcare, Health-Tech, and MedTech is a major advantage; Spanish - a plus; our patients are served in English and Spanish

Условия

The company has Unlimited PTOs Policy and compensated New Years Holidays on top of that. The misuse of the policy isn’t welcomed, though it’s definitely possible to take at least two weeks – and fully compensated – vacation, or more. The company provides Corporate Hardware for employees who completed their Probationary Period, as well as proven their value. We use Google Workspace, Slack, Zoom, and similar collaboration tools for messaging, meetings, and document sharing across the organization. We don’t use Microsoft Outlook, Microsoft Team and similar software. There’s real ability to make most solutions in a Cloud-Native and Third-Party Integrations manner, rarely spinning something self-hosted (e.g., over GKE). Fully Serverless Approach, based on Cloud Functions and Cloud Run is definitely not welcomed. The company partially compensates Team Building events, when multiple teammates are located in nearby countries.