Senior Software Development Engineer in Test (SDET), QA - AI Systems

Nexxa; Toronto; On-site; CAD 120,000 - 160,000; Full time

Job summary

Nexxa is seeking a Senior/Staff SDET to own quality for AI agent systems in industrial environments. You will build evaluation harnesses, simulations, and CI/CD gates, while performing hands-on testing and tracing to ensure safe, effective agent behavior.

Collaborating with ML and backend engineers and Forward Deployed Engineers, you will define what “good” looks like for an agent operating in high-stakes settings, and develop tests, red-team drills, and automated checks to measure performance

Qualifications

• 5+ years as an SDET, QA engineer, or software engineer with testing leadership.
• Manual and automated testing experience, including exploratory testing.
• Strong Python programming and software engineering mindset.
• Experience building test frameworks and infrastructure from scratch.
• Hands-on with CI/CD and containerized environments (Docker/Kubernetes).
• Experience testing APIs and microservices, including end-to-end testing.
• Hands-on testing of LLM-based products, chatbots, or AI agents.
• Understanding of LLM agents, prompting, tool calls, and orchestration.

Responsibilities

• Design, build, and maintain test frameworks, evaluation harnesses, and regression suites for LLM-based agents.
• Create automated pipelines to run evals and tests across models, prompts, and tool integrations within CI/CD.
• Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs.
• Implement API, contract, integration, and end-to-end tests for services and orchestration layers.
• Develop performance, load, and reliability tests for agent workflows; include failure injection.
• Perform hands-on exploratory testing of agent behavior and review traces to judge quality and safety.
• Execute manual regression and release verification in staging and customer-like environments.

Skills

SDET experience

Manual testing

Exploratory testing

Python programming

CI/CD

Docker Kubernetes

API testing

LLM testing

Bug reporting

Tools

GitHub Actions

GitLab CI

Jenkins

Docker

Kubernetes

Job description

ABOUT NEXXA

Nexxa is building the best AI systems for heavy industries, enabling machines, systems and operations to think, decide and act autonomously across manufacturing, large-scale infrastructure, logistics and legacy environments. Our mission is to translate deep technical breakthroughs into operational reality, solving some of the hardest systems-level problems in industry.

ROLE OVERVIEW

We’re looking for a Senior / Staff SDET to own quality for Nexxa’s AI agent systems, combining deep test automation with hands‑on manual and exploratory testing. Our agents plan, call tools, and take multi‑step actions autonomously in industrial environments. You’ll write production‑grade code to build evaluation harnesses, simulation environments, and CI/CD quality gates, and you’ll also get hands‑on with the agents yourself: probing behavior, reviewing traces, and validating releases where automation alone can’t judge whether the agent did the right thing. You’ll work closely with ML engineers, backend engineers, and Forward Deployed Engineers to define what “good” looks like for an agent operating in high‑stakes industrial settings, then build and run the systems and processes that measure it continuously.

KEY RESPONSIBILITIES

• Automation and Infrastructure - Design, build, and maintain test frameworks, evaluation harnesses, and regression suites for LLM‑based agents, covering reasoning quality, tool‑call correctness, task completion, and multi‑turn coherence.
• Build automated pipelines that run evals and test suites on every model, prompt, or tool‑integration change, and integrate them into CI/CD as enforceable quality gates.
• Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs so agents can be tested safely, repeatably, and at scale.
• Implement API, contract, integration, and end‑to‑end tests for the services, tools, and orchestration layers that agents depend on.
• Develop performance, load, and reliability tests for agent workflows, including failure injection and resilience testing.
• Perform hands‑on exploratory testing of agent behavior, including new capabilities, ambiguous or underspecified requests, long multi‑turn sessions, and unusual operational scenarios that scripted tests don’t anticipate.
• Manually review agent traces and outputs to judge reasoning quality, appropriateness of actions, and safety, and use those findings to improve automated checks.
• Execute manual regression and release verification passes in staging and customer‑like environments before new agent capabilities ship.
• Conduct manual red‑teaming sessions (prompt injection, jailbreaks, tool misuse, unsafe actions) in partnership with security teams, then convert successful attacks into automated test cases.
• Support user acceptance testing and pilot rollouts with Forward Deployed Engineers, capturing real‑world failures and feeding them back into test suites.
• Write clear test plans, test charters, and bug reports, and maintain manual test documentation where automation isn’t yet practical or cost‑effective.
• Create and manage golden datasets and labeled test sets, including edge cases, ambiguous inputs, and adversarial prompts specific to industrial and operational contexts, with tooling for versioning and drift detection.
• Design rubrics and run human‑in‑the‑loop evaluations where automated scoring isn’t reliable enough.
• Define and instrument quality metrics beyond simple accuracy (groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations) and build dashboards to track them over time.
• Test agent behavior across the full action loop (planning, tool selection, tool execution, error recovery, and final output), not just the final response.
• Investigate and triage failures where the root cause could be the model, the prompt, the tool/API, or the orchestration logic, and turn them into reproducible, actionable bug reports.
• Establish quality bars, release criteria, and sign‑off processes for new agent capabilities, blending automated gates with manual verification.
• Decide what to automate and what to keep manual, and continually shift repeatable manual work into automation.
• Mentor engineers on testing strategies and test‑driven practices for probabilistic, LLM‑driven systems.
• Advocate for testability, observability, and tracing in agent architecture from day one.

QUALIFICATIONS

• 5+ years as an SDET, QA engineer, or software engineer with a strong testing focus, with demonstrated ownership of test strategy and automation for complex distributed systems.
• Proven experience in both manual and automated testing, including exploratory testing, test planning, and writing high‑quality bug reports.
• Strong programming skills (Python preferred) with a software engineering mindset: clean, maintainable, well‑reviewed code for frameworks, data pipelines, and tooling.
• Experience building test frameworks and infrastructure from the ground up, not just writing test cases within an existing framework.
• Hands‑on experience with CI/CD systems (e.g., GitHub Actions, GitLab CI, Jenkins) and containerized test environments (Docker, Kubernetes).
• Experience testing APIs and microservices, including contract, integration, and end‑to‑end testing.
• Hands‑on experience testing LLM‑based products, chatbots, or AI agents, and an understanding of why traditional deterministic assertions break down for generative systems.
• Practical experience with eval frameworks or tooling (e.g., promptfoo, DeepEval, RAGAS, LangSmith) or a track record of building your own.
• Understanding of how LLM agents work: prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks.
• Experience designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift over time.
• Familiarity with LLM‑specific failure modes: hallucinati