Senior SDET: AI Agentic QA & Automation

Dialpad; Kitchener; Hybrid; CAD 151,000 - 175,000; Full time

Job summary

Dialpad is seeking a Sr. SDET to own test automation and quality frameworks supporting Agentic QA.

You will develop automated tests across frontend UI, backend services, APIs, and audio/text interactions, testing orchestration flows and guardian safeguards to ensure robust coverage. You will build substantial automated test infrastructure, collaborating with the development team to improve testability, stability, and customer delight.

Qualifications

• 6+ years of experience in software engineering or SDET roles with an emphasis on software development.
• Experience testing distributed, cloud-native SaaS systems and APIs.
• Hands-on exposure to AI agents and AI/ML systems (e.g., OpenAI, Claude, Gemini).
• Experience building test frameworks and scalable automation systems.
• Familiarity with AI evaluation techniques (benchmarking, golden datasets).
• Experience with CI/CD pipelines (e.g., Jenkins, GitHub Actions).

Responsibilities

• Own end-to-end quality for agentic features and workflows, including strategy, development, execution, and release qualification.
• Design and build automation tooling and frameworks for AI/LLM-driven systems, including prompt flows, agent orchestration, and tool integrations.
• Develop and maintain evaluation frameworks (evals) to measure response quality, accuracy, and hallucination rates.
• Drive automation coverage (80%+ for critical AI workflows) using deterministic + probabilistic validation approaches.
• Integrate AI quality checks into CI/CD pipelines with fast feedback cycles (<15 minutes for PR validation).
• Build tooling for LLM observability and debugging, including prompt tracing and response analysis.
• Partner with Applied AI teams on prompt engineering, model selection, and evaluation strategies.
• Design and execute performance and load tests for AI services (latency, throughput, cost efficiency).
• Identify and mitigate risks related to hallucinations, bias, safety, and edge cases.
• Define and track AI quality KPIs (task success rates, precision/recall, latency, etc.).
• Participate in design and architecture reviews to ensure systems are testable, observable, and resilient.
• Mentor engineers and contribute to raising the bar on AI quality engineering practices.

Skills

SDET experience

Test automation

Distributed systems

APIs

AI/LLM testing

CI/CD tooling

Frontend testing

Python/JS

Education

Bachelor’s degree in CS or equivalent

Tools

Jenkins

GitHub Actions

LiveKit

OpenAI API

Job description

Dialpad is seeking a Sr. SDET to own test automation and quality frameworks supporting Agentic QA.

You will develop automated tests across frontend UI, backend services, APIs, and audio/text interactions, testing orchestration flows and guardian safeguards to ensure robust coverage. You will build substantial automated test infrastructure, collaborating with the development team to improve testability, stability, and customer delight.