AI Evaluation Engineer — Agentic AI Quality & Benchmarks

Dialpad; Vancouver; Hybrid; CAD 116,000 - 133,000; Full time

Job summary

Dialpad is seeking an AI Evaluation Engineer to join the AI Evaluation team in Vancouver. You will own evaluation coverage for agentic AI systems, co-owning LLM-judge metric development, dataset curation, and structured error analysis to support release-readiness decisions for our voice and chat products.

The role reports to the AI Evaluation manager and requires designing validation strategies, running regression evaluations, and collaborating with applied science, engineering, and Product QA

Qualifications

• Bachelor's or Master’s degree in CS, SE, Computational Linguistics, or related field.
• 3+ years of experience in QA, test engineering, model evaluation, or applied ML quality for AI-driven products.
• Experience designing structured test strategies across manual and automated workflows.
• Comfort working with complex AI systems such as speech, NLP, LLM, or agentic products.
• Experience working with evaluation datasets, gold sets, adversarial test sets, or benchmark creation for AI systems.
• Strong analytical skills for investigating failures, comparing outputs, and identifying actionable quality patterns.
• Experience collaborating with cross-functional technical teams and communicating clearly through documentation and reporting.

Responsibilities

• Design and execute validation strategies for agentic, NLP, and speech workflows across staging, beta, and release candidates.
• Build, run, and improve regression evaluations, A/B comparisons, and red teaming analyses to determine whether product and model changes are ready to move forward.
• Co-own LLM-judge metric development, calibration, and prompt refinement across evaluation dimensions.
• Create, configure, and monitor data annotation jobs to keep evaluation and calibration datasets fed on schedule.
• Develop and maintain QA tooling, notebooks, and pipeline components that make recurring evaluations scalable and reusable across teams.
• Investigate bugs, triage issues, and decide whether problems should become engineering escalations, test set additions, or follow-up analysis.
• Collaborate with cross-functional teams, including applied science, engineering, and Product QA.

Skills

QA testing

Model evaluation

AI/ML quality

NLP/ASR experience

Cross-functional collaboration

Documentation and reporting

Bachelor's or Master's degree in CS/SE

Education

Bachelor's or Master's degree in CS/SE/CL

Tools

QA tooling

Jupyter notebooks

Job description

Dialpad is seeking an AI Evaluation Engineer to join the AI Evaluation team in Vancouver. You will own evaluation coverage for agentic AI systems, co-owning LLM-judge metric development, dataset curation, and structured error analysis to support release-readiness decisions for our voice and chat products.

The role reports to the AI Evaluation manager and requires designing validation strategies, running regression evaluations, and collaborating with applied science, engineering, and Product QA