Senior AI QA Engineer - LLM Testing & Production Readiness

Ellipsis Health; United States; Remote; USD 120,000 - 140,000; Full time

Benefits offered by this job

401(k) matching

Health, vision, and dental insurance

Flexible paid time off

Job summary

Ellipsis Health seeks a Forward Deployed QA Engineer to ensure Sage’s reliability across enterprise workflows. You will bridge QA, AI engineering, and Production Operations, focusing on automated testing, prompt validation, and rapid RCA of LLM behaviors in fast-paced deployments.

This role emphasizes designing robust test cases from client workflows, building automated tests with shadow agents, evaluating prompts, and collaborating with engineering during releases to maintain deployment

Qualifications

• QA engineering experience designing, executing, and managing end-to-end test strategies (Smoke, Sanity, Regression, Functional).
• Hands-on troubleshooting of LLM behaviors, hallucinations, latency, and using LLM tracing tools.
• Ability to write SQL queries (PostgreSQL) and navigate cloud logs (GCP) for rapid RCA.
• Foundational knowledge of Speech-to-Speech pipelines, ASR, audio-to-text workflows, and knowledge-base integrations.
• Strong client-facing skills to manage UAT timelines and coordinate with stakeholders.

Responsibilities

• Analyze client workflows to design robust test cases, including positive and negative scenarios.
• Build automated test scenarios by configuring shadow agents and running tests.
• Evaluate prompts to simulate user behaviors and edge cases within the framework.
• Execute end-to-end testing across staging, pre-production, and production environments.
• Collaborate with engineering during releases to identify and unblock roadblocks.
• Perform daily root-cause analysis on LLM-specific failures (hallucinations, latency, logic issues).
• Debug live production calls and incidents to maintain uptime.
• Validate audio transcripts and S2S pipeline performance across the workflow.
• Apply advanced evaluation frameworks to measure AI quality at scale.
• Understand telephony routing, warm/blind transfers, and voicemail detection in real deployments.

Skills

QA Engineering

LLM & Generative AI

SQL (PostgreSQL)

GCP Logging

Voice AI & Telephony

Telephony Concepts

Client-Facing Communications

Tools

PostgreSQL

GCP

Job description

Ellipsis Health seeks a Forward Deployed QA Engineer to ensure Sage’s reliability across enterprise workflows. You will bridge QA, AI engineering, and Production Operations, focusing on automated testing, prompt validation, and rapid RCA of LLM behaviors in fast-paced deployments.

This role emphasizes designing robust test cases from client workflows, building automated tests with shadow agents, evaluating prompts, and collaborating with engineering during releases to maintain deployment