Senior QA Automation Engineer

IDT; United States; A distancia; USD 120.000 - 160.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Remote work

Annual performance review

Training & conference funding

hardware support

English classes

Referral incentives

Descripción de la vacante

IDT is seeking a QA lead to own the quality strategy for an AI-powered voice product. You will design end-to-end test coverage across language models, speech services, and telephony, prioritizing automation and CI/CD integration.

You will assess variability in model outputs, build robust assertions, and monitor metrics such as accuracy, latency, transcription quality, and escalation correctness. Remote work options available for US-time overlap.

Formación

• 5+ years of QA ownership for a complex, integrated production product.
• Hands-on SIP/VoIP testing experience or related signaling knowledge.
• Strong Python skills to build and maintain automation with pytest.

Responsabilidades

• Own end-to-end QA strategy for an AI-powered voice product.
• Lead test design for multi-turn conversations, interruptions, escalation, and recovery.
• Develop evaluation approaches for variable model outputs and regression detection.
• Test for hallucinations, prompt injection, data exposure, and guardrails.

Conocimientos

QA leadership

Python automation

Test strategy design

Analytical thinking

Agile/Scrum

Herramientas

pytest

Jira

CI/CD pipelines

Descripción del empleo

Role overview

Own quality strategy for an AI-powered voice product that combines language models, speech services, and telephony. This is a manual-testing-heavy leadership role: you will establish coverage standards, design challenging end-to-end scenarios, guide automation priorities, and investigate failures across the conversation and communications stack rather than serving as the primary framework developer.

Responsibilities

• Participate fully in Scrum ceremonies and maintain clear testing artifacts and traceability.
• Define end-to-end test strategy for multi-turn conversations, interruptions, ambiguous requests, escalation, recovery, and out-of-scope behavior.
• Build evaluation approaches for variable model output using representative datasets, semantic or rubric-based assertions, and regression detection.
• Test for hallucinations, weak grounding, prompt injection, guardrail bypasses, and unauthorized exposure of customer information.
• Provision SIP endpoints and call scenarios, then isolate failures across language models, speech recognition, speech synthesis, media transport, and routing.
• Expand automated coverage, connect suites to CI/CD release gates, and report metrics such as task completion, intent accuracy, latency, transcription quality, and escalation correctness.

Requirements

• At least five years of QA experience owning quality for a complex, integrated production product.
• Practical SIP/VoIP testing experience, including signaling, call flows, media handling, softphones, or trunking; able to use traces or packet captures.
• Strong Python skills sufficient to build and maintain automation with tools such as pytest, plus experience reviewing and debugging test suites.
• Experience testing probabilistic or non-deterministic systems and defining meaningful assertions for variable results.
• Strong analytical and scenario-design skills, with the ability to use quality metrics to support release decisions.
• Hands-on experience with formal test management systems and Jira, plus professional spoken and written English with overlap with US Eastern hours.

Nice to have

• Experience with conversational AI, LLM evaluation, speech quality, retrieval grounding, WebRTC or media-layer testing, IVR/contact-center platforms, multilingual testing, or prompt analysis.

Benefits and work setup

• Remote work, B2B engagement, long-term growth, annual performance review, hardware support, professional training and conference funding, English classes, and referral incentives.