Software Development Engineer in Test - Agentic Automation

Jobgether; Canada; A distancia; CAD 120.000 - 170.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Equity opportunities

Wellness programs

Vacations and holidays

Parental leave

Descripción de la vacante

Jobgether in Canada is seeking an experienced Software Development Engineer in Test (SDET) to shape how AI agents are tested and trusted in production. You will join a platform engineering team within a large-scale cloud cybersecurity environment, building automation for reliability, performance, and quality.

You will work on agentic test generation, AI-powered workflows, and eval-driven development, collaborating across the lifecycle in a remote-first, autonomous team culture.

Formación

• 5+ years in software quality with 3+ years SDET or test automation.
• Strong programming in Java, Node.js, or Python; Python preferred for AI eval.
• Experience with CI/CD pipelines (Jenkins or GitLab CI).
• Experience testing web services and APIs (REST, gRPC, JSON).
• Experience with performance/load testing (Locust, k6).
• Skeptical, verification-first approach to AI outputs.
• Experience applying AI to improve workflows and outcomes.

Responsabilidades

• Design, develop, and maintain automated test suites for distributed, cloud-based systems.
• Maintain and extend agentic test generation and analysis within core testing frameworks.
• Build AI-powered workflows and agent skills to reduce manual testing effort.
• Maintain evaluation suites, golden datasets, and scoring pipelines to assess AI agents.
• Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration.
• Identify and mitigate unreliable AI outputs, including hallucinated assertions and flaky tests.
• Design and execute performance, load, and stress tests to identify bottlenecks.
• Test integrations across platform systems, services, data, applications, and customer-facing features.
• Review product, system, and agent designs to assess testability and coverage.
• Share technical knowledge and contribute to quality standards.

Conocimientos

Java

Node.js

Python

CI/CD

REST APIs

Performance testing

Distributed systems

Team collaboration

Educación

BSc in Computer Science

Herramientas

Jenkins

GitLab CI/CD

Locust

k6

Grafana

Prometheus

CloudWatch

Docker

Kubernetes

MongoDB

Descripción del empleo

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Development Engineer in Test - Agentic Automation based in Canada.

This is an opportunity for an experienced SDET to help shape how AI agents are tested, evaluated, and trusted in production. You will join a platform engineering team working on a large-scale, cloud-based cybersecurity environment, building the automation that supports reliability, performance, and quality. Alongside traditional functional and integration testing, you will contribute to agentic test generation, AI-powered workflows, and eval-driven development. Your work will help ensure that AI systems behave accurately, safely, and consistently before reaching production. You will collaborate with engineers across the development lifecycle in a remote-first environment that values autonomy, experimentation, and continuous improvement.

Accountabilities

• Design, develop, and maintain automated test suites and frameworks for distributed, cloud-based systems.
• Maintain and extend agentic test generation and analysis capabilities within core testing frameworks.
• Build AI-powered workflows and agent skills that reduce manual testing effort, including test data setup, failure triage, and release validation.
• Maintain evaluation suites, golden datasets, and scoring pipelines to assess AI agents across correctness, tool usage, execution trajectories, faithfulness, safety, latency, and cost.
• Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration before release, incorporating production failures into future evaluations.
• Identify and mitigate unreliable AI outputs, including hallucinated assertions, flaky or redundant tests, inaccurate failure analysis, and evaluation bias.
• Design and execute performance, load, and stress tests to identify bottlenecks and validate the scalability of distributed services.
• Test integrations across platform systems, services, data, applications, and customer-facing features.
• Review product, system, and agent designs to assess testability, evaluation coverage, and opportunities for improvement.
• Share technical knowledge and contribute to team-wide quality and evaluation standards.

Requirements

• Degree in Computer Science or equivalent practical experience, with a solid understanding of object-oriented programming, data structures, algorithms, and distributed systems.
• At least 5 years of experience in software quality, including 3+ years of hands-on SDET or test automation development.
• Strong programming skills in at least one of Java, Node.js, or Python; Python is preferred for AI evaluation and agent-related work.
• Demonstrated experience designing, developing, and maintaining test frameworks integrated into CI/CD pipelines such as Jenkins or GitLab CI/CD.
• Experience testing web services and APIs, including REST, gRPC, and JSON.
• Hands-on performance and load testing experience with tools such as Locust or k6, including workload design and interpretation of latency percentiles, throughput, and error rates.
• A skeptical, verification-first approach to AI-generated output, with the ability to distinguish when deterministic assertions are appropriate and when probabilistic evaluation is needed.
• Experience applying AI technologies to improve decision-making, streamline workflows, increase efficiency, and deliver business outcomes.
• Strong analytical, problem-solving, and collaboration skills, with the ability to communicate technical findings and contribute to shared quality standards.

Nice to have

• End-to-end eval-driven development experience, including release gating, trajectory and tool-use evaluations, and calibrating LLM-as-judge systems against human review.
• Experience with prompt engineering and designing agent context or tools.
• Experience integrating performance tests into CI/CD pipelines and using observability tools such as Grafana, Prometheus, or CloudWatch.
• Background in cybersecurity, threat intelligence, malware analysis, or sandboxing and detonation environments.
• Familiarity with technologies such as MongoDB, MinIO, RabbitMQ, Kafka, AWS services (EC2, S3, DynamoDB), Docker, Nomad, Terraform, Unix/Linux, Git, Jenkins, GitLab CI/CD, Jira, and Bitbucket.
• Experience with LLM APIs, agent frameworks, and LLM evaluation tooling.

Benefits

• Annual base salary ranging from CA$120,000 to CA$170,000, plus variable or incentive compensation, equity, and benefits.
• Market-leading compensation and equity award opportunities.
• Comprehensive physical and mental wellness programs.
• Competitive vacation allowance and holidays.
• Paid parental and adoption leave.