Senior QA Engineer — AI Agent Systems & Autonomy

Engg; Toronto; On-site; CAD 100,000 - 150,000; Full time

Job summary

Nexxa.AI in Toronto seeks a Lead/Senior/Staff QA Engineer to own quality for AI agent systems that plan, call tools, and act in industrial environments. You will design evaluation frameworks for non-deterministic systems, build golden datasets, and stress-test agent behavior across planning, tool use, and recovery.

You will collaborate with ML and backend teams, develop robust testing pipelines, and mentor others on probabilistic system testing, ensuring observability and strong quality bars

Qualifications

• 5+ years in QA/SDET roles owning test strategy for complex systems.
• Hands-on experience testing LLM-based products, chatbots, or AI agents.
• Practical experience with eval frameworks or building your own.
• Strong scripting ability (Python) for test automation and data pipelines.
• Understanding of LLM agents: prompting, tool calls, context management, and orchestration.

Responsibilities

• Design and build evaluation harnesses and regression suites for LLM-based agents, covering reasoning quality, tool-call correctness, task completion, and multi-turn coherence.
• Develop golden datasets and labeled test sets, including edge cases and adversarial prompts.
• Define and track quality metrics beyond accuracy—groundedness, hallucination rate, task success, latency/cost tradeoffs, safety violations.
• Build automated pipelines to run evals on every model, prompt, or tool integration change, and integrate into CI/CD.
• Conduct structured red-teaming and adversarial testing with security teams.
• Test agent behavior across the full action loop—planning, tool execution, error recovery, and final output.
• Triage failures and translate eval failures into actionable bug reports.
• Establish quality bars and sign-off criteria for new agent capabilities before customers see them.
• Mentor other engineers on testing probabilistic, LLM-driven systems.
• Advocate for testability and observability from day one.

Skills

QA Ownership

LLM Testing

Python Scripting

Eval Frameworks

Prompting

Ambiguity Handling

Written Communication

Tools

promptfoo

DeepEval

RAGAS

LangSmith

Job description

Nexxa.AI in Toronto seeks a Lead/Senior/Staff QA Engineer to own quality for AI agent systems that plan, call tools, and act in industrial environments. You will design evaluation frameworks for non-deterministic systems, build golden datasets, and stress-test agent behavior across planning, tool use, and recovery.

You will collaborate with ML and backend teams, develop robust testing pipelines, and mentor others on probabilistic system testing, ensuring observability and strong quality bars