Senior AI Voice QA Engineer - Non-Deterministic Testing

Lever, Inc.; Santiago; Presencial; CLP 40.000.000 - 60.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Remote work opportunity

Competitive salary

Professional growth

Equipment provided

Descripción de la vacante

IDT Technologies is seeking a Senior QA Engineer to own quality for our AI voice virtual agent, spanning LLM, speech, and telephony layers. You will design test strategies for non-deterministic systems and validate complex conversational behaviour.

Responsibilities include embedding in a Scrum team, building an evaluation framework, and automating coverage from SIP to model outputs, with a focus on release-quality metrics and rigorous traceability.

Formación

• 5+ years in QA with complex, integrated product.
• Practical experience testing SIP/VoIP systems.
• Demonstrated creativity in test design for live scenarios.
• Proficiency in Python for automation frameworks.
• Experience testing non-deterministic or probabilistic systems.
• Strong analytical skills for metrics and release decisions.
• Hands-on experience with Jira and TMS tools.

Responsabilidades

• Embed fully within a Scrum team and participate in ceremonies.
• Own the end-to-end test strategy for the AI voice virtual agent across LLM, speech, and telephony layers.
• Design test scenarios for complex conversational behaviour (multi-turn, interruptions, barge-in, incomplete input, escalation).
• Build and maintain an evaluation framework for non-deterministic output (golden data, rubric-based assertions, regression detection).
• Adversarially test the agent against hallucinations, grounding issues, prompt injections, and data disclosure risks.
• Prepare and provision SIP devices, endpoints, and call scenarios to automate calls end-to-end.
• Diagnose failures across the stack, from LLM to media path and call routing.
• Extend automated coverage and integrate test suites into CI/CD pipelines as release gates.
• Define and track core quality metrics (task completion, containment, accuracy, latency, escalation).
• Maintain documented test coverage in TMS with traceability to requirements.

Conocimientos

QA expertise

Python scripting

SIP/VoIP testing

Test design

English proficiency

Herramientas

pytest

Playwright

JsSIP

Descripción del empleo

IDT Technologies is seeking a Senior QA Engineer to own quality for our AI voice virtual agent, spanning LLM, speech, and telephony layers. You will design test strategies for non-deterministic systems and validate complex conversational behaviour.

Responsibilities include embedding in a Scrum team, building an evaluation framework, and automating coverage from SIP to model outputs, with a focus on release-quality metrics and rigorous traceability.