Senior AI Quality & Red Team Engineer

Netskope; Taguig; On-site; PHP 7,421,150 - 11,131,725; Full time

Job summary

Netskope is seeking a Senior AI Quality & Red Team Engineer to lead testing, stress-testing, and breaking AI agents before they reach production. You will own the automated harness that scales across a growing fleet, designed to run unattended and be maintenance-free.

You will design adversarial test scenarios, automate them across changes, and define pass criteria for each agent, ensuring safety and compliance.

Qualifications

• At least 4 years in software quality, security testing, or related discipline, with 1–2 years specifically evaluating or red-teaming LLM-based systems.
• Strong Python skills; evaluation harness, adversarial test scripts, and automated pipelines built in Python.
• Working knowledge of REST APIs and webhook/event-driven patterns to build test harnesses that call an agent's tools directly.
• Experienced in building automated test infrastructure and integrating it into CI/CD, focusing on pipelines and repeatability.
• Hands-on familiarity with at least one adversarial testing or LLM evaluation tool (DeepTeam, Garak, PyRIT, Promptfoo) and mapping to OWASP LLM Top 10 or NIST AI risk framework.
• Understanding of prompt injection, jailbreaking, and sycophancy failure modes.

Responsibilities

• Build and maintain the automated evaluation suite that every agent runs against before production, designed to scale across a growing fleet.
• Design adversarial test scenarios—prompt injection attempts, sycophancy checks, multi-attempt attacks—and automate them for every relevant change.
• Own the break-it-on-purpose pass for new agents, ensuring they cannot expose data or operate outside tool boundaries; automate this as part of the pipeline.
• Collaborate with Data Steward on data sensitivity classifications to reflect real risk in tests.
• Define per-agent capability passes and automate thresholds to keep evaluation aligned with fleet growth.
• Maintain guardrail and negative-test catalogs (fail-closed vs fail-graceful) and add new cases as failures are discovered.
• Produce audit-ready evidence for every agent evaluation, generated automatically as part of the pipeline.
• Track drift across the fleet over time to catch slow-moving issues across hundreds of agents.

Skills

Python

REST APIs

CI/CD pipelines

Adversarial testing

LLM evaluation

Tools

DeepTeam

Garak

PyRIT

Promptfoo

Job description

Netskope is seeking a Senior AI Quality & Red Team Engineer to lead testing, stress-testing, and breaking AI agents before they reach production. You will own the automated harness that scales across a growing fleet, designed to run unattended and be maintenance-free.

You will design adversarial test scenarios, automate them across changes, and define pass criteria for each agent, ensuring safety and compliance.