Senior QA Automation Engineer - SIP/VOIP

IDT

United States

Remoto

USD 110.000 - 165.000

Tempo pieno

14 giorni+

Aumenta le possibilità di ricevere una risposta da questo datore di lavoro

Il tuo curriculum e una lettera di presentazione personalizzati, che corrispondono esattamente a ciò che sta cercando.

Supera i filtri ATS

Vantaggi offerti da questo lavoro

Remote work opportunity
Competitive salary
Career growth
Training & conferences
Referral program
English classes
Hardware provided

Descrizione del lavoro

IDT Technologies is seeking an experienced Senior QA Engineer to own quality for an AI voice virtual agent. You will design tests for non-deterministic behavior, include SIP/VoIP testing, and drive evaluation across LLM, speech, and telephony layers.

Remote work from the US is offered with a strong emphasis on automation and CI/CD integration. You will embed within a Scrum team, own end-to-end test strategy, and develop frameworks to measure metrics like task completion, transcription accuracy,

Competenze

  • 5+ years in QA and hands-on ownership of a complex product.
  • Experience testing SIP/VoIP systems and call flows.
  • Creativity in test design for live scenarios.
  • Proficiency in Python to build automation frameworks.
  • Experience testing non-deterministic systems with robust assertions.
  • Strong analytical skills to define and justify quality metrics.
  • Hands-on with TMS tools (Allure/JA) and Jira.
  • Professional fluency in English, with US Eastern overlap.

Mansioni

  • Embed fully within a Scrum team and own end-to-end test strategy for AI voice agent.
  • Design test scenarios for multi-turn conversations, interruptions, and escalation.
  • Build an evaluation framework for non-deterministic output and regression detection.
  • Adversarially test the agent for hallucinations, grounding, and prompt injections.
  • Prepare SIP devices and call scenarios to automate end-to-end origination.
  • Diagnose failures across LLM, ASR, TTS, media path, and routing layers.
  • Extend automated coverage and integrate tests into CI/CD as release gates.
  • Define core quality metrics: task completion, accuracy, latency, escalation.

Conoscenze

QA
Python
SIP/VoIP testing
Test design
Analytics
English fluency

Strumenti

pytest
Playwright
JsSIP/WebRTC
SIP softphones
Allure TestOps
Jira
Jenkins & GitHub Actions
Grafana
LLM evaluation tooling

Descrizione del lavoro

We are looking for an experienced Senior QA Engineer to own quality for our AI voice virtual agent. This role is for someone who is equally comfortable reasoning about a large language model's behavior and reading a SIP trace.

You should be interested in testing non-deterministic systems. A voice agent takes a live phone call, transcribes speech that may be accented or noisy, interprets intent, reasons over an LLM, generates a spoken reply, and decides whether to resolve the request or hand it to a human—all within a latency budget callers will notice. Nothing in that chain can be validated with a simple assertion of equality, so you will design the evaluation approach that makes it testable anyway.

You should also be interested in the underlying telephony. Testing this agent means placing real calls: provisioning SIP endpoints and softphones, driving calls programmatically, controlling media conditions, and distinguishing a model failure from a codec, jitter, or call-routing issue.

Responsibilities
  • Embed fully within a Scrum team, participating in all ceremonies and maintaining associated artifacts.

  • Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers.

  • Design test scenarios for complex conversational behavior, including multi-turn dialogues, interruptions and barge-in, ambiguous or incomplete input, out-of-scope requests, human escalation/transfer, and failure recovery.

  • Build and maintain an evaluation framework for non-deterministic output, leveraging golden datasets, semantic/rubric-based assertions, and automated regression detection for prompt, model, or retrieval changes.

  • Adversarially test the agent against hallucinations, incorrect grounding, guardrail bypasses, prompt injections, and unauthorized disclosure of customer data.

  • Prepare and provision SIP devices, endpoints, and call scenarios to automate call origination end-to-end.

  • Diagnose failures across the entire stack, isolating issues down to the LLM, ASR (speech recognition), TTS (text-to-speech), media path, or call routing layer.

  • Extend automated coverage and integrate test suites into the CI/CD pipeline to serve as release gates.

  • Define and track core quality metrics, including task completion, containment, intent accuracy, transcription accuracy, response latency, and escalation correctness.

  • Maintain documented test coverage in the TMS with full traceability to business requirements.

  • Communicate and communicate quality risks to stakeholders backed by data and evidence.

Requirements:
  • 5+ years in QA, with recent hands-on ownership of a complex, integrated product.

  • Practical experience testing SIP/VoIP systems, including call flows, SIP signaling, media handling, softphones, or SIP trunking, with the ability to troubleshoot using SIP traces or packet captures.

  • Demonstrated creativity in test design (live scenario design will be a core part of the interview process and heavily weighted).

  • Proficiency in Python, sufficient to independently build and maintain automation frameworks.

  • Experience testing non-deterministic or probabilistic systems, with a clear approach to designing assertions for variable outputs.

  • Strong analytical skills, including the ability to define quality metrics, justify their calculation, and use data to back release decisions.

  • Hands-on experience with formal TMS tools and Jira.

  • Professional fluency in English (spoken and written), with reliable overlap with US Eastern business hours.

We use the following technologies:
  • Python with pytest for test automation.

  • Playwright for browser and WebRTC-based call driving.

  • JsSIP / WebRTC for programmatic call origination and media control.

  • SIP softphones and devices for manual and exploratory testing.

  • Allure TestOps as our TMS, linking automated runs to documented test cases.

  • Jira for project management and issue tracking.

  • Jenkins & GitHub Actions for CI/CD pipelines.

  • Grafana for quality and delivery metrics.

  • LLM APIs and evaluation tooling — currently in active development, and we are open to any effective approach.

Pluses:
  • Direct experience testing LLM-based or conversational AI products, including evaluation frameworks and LLM-as-a-judge approaches.

  • Familiarity with speech technologies, including ASR/transcription accuracy measurement (WER), TTS quality assessment, endpointing, and silence detection.

  • Experience with RAG or retrieval-grounded systems, specifically in evaluating grounding correctness and context retrieval.

  • WebRTC and media-layer testing experience, including codec behavior and performance degradation under jitter or packet loss.

  • Background in testing IVR, contact center, or conversational IVR platforms.

  • Multilingual testing capabilities, with emphasis on Spanish/English code-switching.

  • Literacy in prompt engineering, sufficient to analyze system prompts and identify potential failure modes

We are pleased to offer:
  • Remote work opportunity!

  • B2B Employment ($, gross).

  • Stable job with long-term growth perspective.

  • Competitive salary with annual performance review.

  • Really good hardware.

  • An exciting and challenging job with talented people around.

  • Continuous learning and career growth opportunities.

  • Compensation for professional training, seminars, and conferences.

  • Referral program: get rewarded for helping us grow the team with talented people.

  • Company-supported English classes to enhance your professional growth.

Creativity is a core requirement for this role, not a nice-to-have. The defects that matter here are found by someone who imagines the caller who interrupts mid-sentence, switches languages halfway through, gives a partial account number, answers a different question, or deliberately tries to jailbreak the agent. Scenario design is the primary skill we are hiring for.

Current State: We have automated call testing built on JsSIP/WebRTC with Python and Playwright, alongside a growing body of test scenarios. The evaluation layer—how we score responses consistently, catch regressions when a prompt or model changes, and report quality metrics leadership can act on is where this role will have the most impact.

About us:

IDT Technologies is a part of IDT Corporation—an American telecommunications company founded in 1990 and headquartered in New Jersey. It is an industry leader in prepaid communication and payment services and one of the world's largest international voice carriers. We are listed on the NYSE, employ over 1,500 people across 20+ countries, and have revenues in excess of $1.5 billion.

Net2Phone (https://www.net2phone.com/) is a global provider of unified cloud communications solutions. It offers an integrated communications suite featuring powerful collaboration tools, including voice, messaging, chat, and unlimited domestic and international calling to 23+ countries.

Ottieni la revisione del curriculum gratis e riservata.

o trascina qui il file.

Similar jobs

Offerte di lavoro simili che vale la pena confrontare

Senior QA Automation Engineer
Senior QA Automation Engineer

IDT Corporation • Stati Uniti

Remoto
USD 120.000 - 160.000
Remote work opportunity
Senior QA Automation Engineer
Senior QA Automation Engineer

IDT • New Jersey

Remoto
MXN 700.000 - 1.100.000
Remote work opportunity!
B2B Employment ($, gross)
Competitive salary with annual review
+1
Senior QA Automation Engineer
Senior QA Automation Engineer

IDT • Stati Uniti

Remoto
USD 120.000 - 160.000
Remote work
Annual performance review
Training & conference funding
+3
Remote: Senior QA Automation Engineer, SIP/VoIP & AI Voice
Remote: Senior QA Automation Engineer, SIP/VoIP & AI Voice

IDT • Stati Uniti

Remoto
USD 110.000 - 165.000
Remote work opportunity
Competitive salary
Career growth
+4
Remote Senior QA Engineer – AI Voice, LLM & SIP Testing
Remote Senior QA Engineer – AI Voice, LLM & SIP Testing

IDT Corporation • Stati Uniti

Remoto
USD 120.000 - 160.000
Remote work opportunity
Software Engineer, Test & Infrastructure II (Bilingual Spanish)
Software Engineer, Test & Infrastructure II (Bilingual Spanish)

Vail Systems Inc. • Illinois

Ibrido
USD 89.400 - 116.400
Multiple medical, dental, and vision plan options
401(k) savings plan with company match
35 days total annual PTO
+3
Remote Senior QA for AI Voice & SIP Automation
Remote Senior QA for AI Voice & SIP Automation

IDT • New Jersey

Remoto
MXN 700.000 - 1.100.000
Remote work opportunity!
B2B Employment ($, gross)
Competitive salary with annual review
+1
Senior Software Engineer - Conversational AI Platform
Senior Software Engineer - Conversational AI Platform

Car Wars • Abilene (TX)

In loco
USD 130.000 - 180.000
Full benefits package
401k with company match
Monthly gym reimbursement
+4
Quality Assurance Engineer
Quality Assurance Engineer

Napster • Stati Uniti

Ibrido
USD 120.000 - 170.000
Senior QA Engineer
Senior QA Engineer

Vi • Stati Uniti

Remoto
USD 120.000 - 180.000