Remote Senior QA for AI Voice & SIP Automation

Lever, Inc.; New Jersey; On-site; MXN 700,000 - 1,100,000; Full time

Benefits offered by this job

Remote work opportunity!

B2B Employment ($, gross)

Competitive salary with annual review

Company-sponsored English classes

Job summary

IDT Technologies in Mexico is seeking a Senior QA Engineer to own the quality of an AI voice virtual agent, combining LLM reasoning with SIP/VoIP testing across the full stack. The role emphasizes designing test scenarios for complex conversations and automating evaluation of non-deterministic outputs.

You will drive end-to-end testing, from data-driven evaluation to latency and escalation metrics, while ensuring robust telemetry and release-grade quality in a fast-paced telephony environment.

Qualifications

• 5+ years in QA
• Practical experience testing SIP/VoIP systems
• Demonstrated creativity in test design
• Proficiency in Python for automation
• Experience testing non-deterministic or probabilistic systems
• Strong analytical skills with data-driven decision making
• Hands-on experience with Jira and TMS tools
• Fluent English (spoken and written) with overlap to US Eastern hours

Responsibilities

• Embed fully within a Scrum team and own the end-to-end test strategy for the AI voice virtual agent across LLM, speech, and telephony layers.
• Design test scenarios for complex conversational behavior including multi-turn dialogues, interruptions, and escalation paths.
• Build and maintain an evaluation framework for non-deterministic output and regression detection for prompts, models, or retrieval changes.
• Adversarially test the agent against hallucinations, grounding issues, prompt injections and data disclosure risks.
• Prepare and provision SIP devices and call scenarios to automate call origination end-to-end.
• Diagnose failures across LLM, ASR, TTS, media path, and call routing layers.
• Extend automated coverage and integrate test suites into CI/CD as release gates.
• Define and track core quality metrics and maintain test coverage traceability.

Job description

IDT Technologies in Mexico is seeking a Senior QA Engineer to own the quality of an AI voice virtual agent, combining LLM reasoning with SIP/VoIP testing across the full stack. The role emphasizes designing test scenarios for complex conversations and automating evaluation of non-deterministic outputs.

You will drive end-to-end testing, from data-driven evaluation to latency and escalation metrics, while ensuring robust telemetry and release-grade quality in a fast-paced telephony environment.