AI Evaluation Engineer - Agentic AI Quality & Metrics

Doist; Kitchener; On-site; CAD 104,000 - 120,000; Full time

Benefits offered by this job

Ontario base salary

Job summary

Dialpad is seeking an AI Evaluation Engineer to own evaluation coverage for its agentic AI systems. The role focuses on co-owning LLM-judge metrics, calibration, and dataset curation to support release-readiness for voice and chat solutions.

The position reports to the AI Evaluation manager and may be based in our Vancouver office. The ideal candidate has a CS/SE/linguistics background with 3+ years in QA, model evaluation, or applied ML quality for AI products.

Qualifications

• Bachelor's or Master's degree required in CS/SE/Computational Linguistics or related field.
• 3+ years of QA, test engineering, model evaluation, or applied ML quality for AI-driven products.
• Experience designing structured test strategies across manual and automated workflows.
• Comfort with complex AI systems such as speech, NLP, LLM, or agentic products.
• Experience with evaluation datasets, gold sets, adversarial test sets, or benchmark creation for AI systems.
• Strong analytical skills for investigating failures, outputs comparison, and pattern identification.
• Experience collaborating with cross-functional technical teams and clear documentation/reporting.

Responsibilities

• Design and execute validation strategies for agentic, NLP, and speech workflows across staging, beta, and release candidates.
• Build, run, and improve regression evaluations, A/B comparisons, and red-teaming analyses for readiness of product and model changes.
• Co-own LLM-judge metric development, calibration, and prompt refinement across evaluation dimensions.
• Create, configure, and monitor data annotation jobs to feed evaluation/calibration datasets on schedule.
• Develop and maintain QA tooling, notebooks, and pipeline components for scalable, reusable evaluations across teams.
• Investigate bugs, triage issues, and decide on engineering escalations, test-set additions, or follow-up analysis.
• Collaborate with cross-functional teams including applied science, engineering, and Product QA.

Skills

QA

Test engineering

Model evaluation

Applied ML quality

Structured test strategies

Speech/NLP/LLM

Evaluation datasets

Adversarial test sets

Cross-functional collaboration

Documentation & reporting

Education

Bachelor's or Master's degree in Computer Science, Software Engineering, Computational Linguistics, or a related field

Job description

Dialpad is seeking an AI Evaluation Engineer to own evaluation coverage for its agentic AI systems. The role focuses on co-owning LLM-judge metrics, calibration, and dataset curation to support release-readiness for voice and chat solutions.

The position reports to the AI Evaluation manager and may be based in our Vancouver office. The ideal candidate has a CS/SE/linguistics background with 3+ years in QA, model evaluation, or applied ML quality for AI products.