Remote Senior QA Engineer - AI Voice & LLM Testing

VI; United States; Remote; USD 110,000 - 170,000; Full time

Job summary

VI is seeking a Senior QA Engineer to own quality for voice agents and LLM-based workflows in a production health-system context. You will design test plans, execute regression suites, and validate transcripts, latency, and logs across speech-to-text, model, and TTS layers.

You will set pass criteria for releases, build repeatable tests for non-deterministic outputs, and wire tooling into CI. A strong background in Python automation and a system-thinking mindset are essential.

Qualifications

• Hands-on QA of production voice agents and LLMS in a real environment.
• Proven ability to diagnose failures across speech-to-text, model, and TTS layers.
• Experience turning vague failures into reproducible tickets for engineers.
• Experience with non-deterministic outputs and establishing test tolerances.

Responsibilities

• Own voice agent testing, including transcripts, latency traces, and logs.
• Recommend improvements to test strategies and regression suites.
• Define release criteria and ensure QA gates before client deployment.
• Develop and maintain CI-integrated QA tooling for automation.
• Identify and mitigate guardrail and security risks in production.

Skills

QA on voice agents

LLM systems testing

Python testing automation

Non-deterministic output handling

System-wide QA thinking

Tools

CI tooling

Job description

VI is seeking a Senior QA Engineer to own quality for voice agents and LLM-based workflows in a production health-system context. You will design test plans, execute regression suites, and validate transcripts, latency, and logs across speech-to-text, model, and TTS layers.

You will set pass criteria for releases, build repeatable tests for non-deterministic outputs, and wire tooling into CI. A strong background in Python automation and a system-thinking mindset are essential.