Senior AI Evaluation Engineer - Agentic Systems

Dayforce

Toronto

Hybrid

CAD 120,000 - 180,000

Full time

3 days ago

Be an early applicant

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Job summary

Dayforce is seeking an AI Engineer - Agentic Systems Evaluation to help define how we measure, test, and improve the quality of enterprise AI systems. You will design evaluation frameworks for agentic AI, multi-agent workflows, and RAG-driven systems, tackling reliability, cost, and safety.

Join a team building scalable evaluation infrastructure and producing rigorous benchmarks to guide production deployments.

Qualifications

Experience with Python and software engineering.
LLM application development experience.
AI/ML evaluation and experimentation.
Automated testing and data analysis experience.
Familiarity with RAG, embeddings, hybrid search, and reranking.
Experience with tool/function calling and structured outputs.
Agent orchestration and multi-agent workflows.
AI observability and tracing.
APIs and enterprise integrations.

Responsibilities

Build Agentic Evaluation Frameworks covering the full AI execution lifecycle.
Design experiments and benchmarks for architecture comparisons (single-agent vs multi-agent).
Develop automated evaluation and regression testing infrastructure.
Analyze agent traces, tool usage, and failure modes to improve reliability and cost.