How to Use Red Teaming AI: A Practical Walkthrough

Why AI Red Teaming Became Non-Negotiable (and Why Most Teams Haven't Started)

The numbers are hard to ignore: a multi-turn jailbreak succeeds 97% of the time within just five conversation turns, according to attack data compiled by Vectra AI. Meanwhile, HackerOne recorded a 540% rise in valid prompt injection reports in 2025 alone. AI systems aren't just theoretically vulnerable — they're being broken at scale, right now.

What makes this worse is how few organizations are responding. Only 22% of organizations conducted adversarial AI testing in 2025, per a market analysis by Dataintelo, even as the attack surface exploded. That gap between threat and defense is where breaches happen.

The regulatory clock is also ticking. Article 55 of the EU AI Act, in effect since August 2025, requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing. Once the AI Office's enforcement powers begin in August 2026, red teaming stops being a discretionary safety exercise and becomes an auditable deliverable regulators can request on demand. Teams that haven't started will be scrambling to produce evidence they simply don't have.

The shift from chat assistants to autonomous agents makes the problem more urgent, not less. Every enterprise that connects a model to email, code repositories, or payment systems through agents and MCP servers adds attack paths that traditional application testing doesn't cover. Agent attacks follow the adversary across multiple tool calls — something a conventional penetration test was never designed to catch.

The NVIDIA AI Red Team, after six months of agent assessments, documented just how exposed these systems are:

• Social engineering — an attacker claiming to be "debugging" or an "admin user" — proved remarkably effective, with one agent writing and executing a reverse shell on request.
• "Frog-boiling" attacks used progressive manipulation across multiple interactions to convince agents into revealing secrets.
• Malicious instructions hidden in legitimate workflows, like a standard-looking package install, bypassed agent resistance entirely.
• File-write tools alone could yield remote code execution by writing to shell config or skills files.

That last point deserves attention from anyone building with agent skills. NVIDIA's team concluded that prompt-based guardrails, including the LLM-as-a-judge pattern, don't close these gaps — architectural controls do. And since skills files are a known write-target for code execution, verifying what you attach to an agent before you attach it matters. That's part of why SkillzBench is building free security scanning and benchmarking for agent skills, alongside a free MCP connector that only exposes the skills and tools you explicitly choose.

The message for teams is blunt: attackers have already adapted, regulators have already scheduled the deadline, and most organizations haven't run a single adversarial test. Closing that gap starts with understanding what red teaming actually involves — and it's more approachable than most teams assume.

Start With a Threat Model, Not a Tool

The most common mistake in AI red teaming is opening a tool before writing a threat model. CSET researchers, who surveyed 23 red-teaming resources, call the written threat model "the core design feature of a red-teaming exercise that influences all other design choices" — meaning every later decision, from tool selection to scoring, flows from it. A threat model is simply a description of the AI system, the relevant vulnerabilities, and the contexts in which they might arise, including human interactions.

Start by answering three questions in writing. What system are you testing — a chatbot, a code assistant, or an autonomous agent connected to tools? What could an attacker realistically want from it — data, code execution, or unsafe content? And who interacts with it, in what context? A model wired into email or payment systems carries very different risks than a customer-facing Q&A bot, and testing with red team agents is the fastest-moving edge of AI security precisely because agent attacks follow threats across multiple tool calls.

Once the threat model exists, scope the tests using established frameworks rather than improvising. Practitioners recommend layering several standards:

• OWASP Top 10 for LLMs — defines what to test, from prompt injection to supply chain risks
• OWASP GenAI Red Teaming Blueprint — defines how to test, across planning, execution, and post-engagement phases
• NIST AI RMF — aligns testing with broader risk management obligations, including the EU AI Act's documented adversarial testing requirement
• MITRE ATLAS — maps real-world adversarial techniques against AI systems

Why does the threat model come first? Because tools embed their own assumptions. CSET warns that a tool whose threat model doesn't match yours "may be unhelpful or even misleading" — it will happily scan for the wrong things and report green while real risks go untested. The stakes are real: a vendor analysis reports multi-turn jailbreaks succeed 97% of the time within five conversation turns, so a tool that only tests single-turn prompts misses the most effective attack class.

This matters especially for agent builders choosing what to connect. NVIDIA's AI Red Team found that file-write tools alone can yield remote code execution when an agent writes to configuration or skills files — and that least-privilege tools, MCPs, and skills are a core architectural defense. SkillzBench is building free security scanning and version-pinned verification of agent skills, plus a free MCP connector with per-agent loadouts, so you can scope what each agent sees before testing it.

Write the threat model down, scope with frameworks, and only then pick a tool. Everything else in the walkthrough depends on getting this order right.

Run a Hybrid Attack: Automated Scans Plus Human Creativity

Automation alone will not find everything — and the numbers prove it. A Stanford benchmark from December 2025 found that the best fully autonomous agent missed a critical vulnerability that 80% of human testers caught, which is why mature programs run a hybrid attack: broad automated coverage first, human creativity second.

Start with open-source tooling. Microsoft's PyRIT, NVIDIA's Garak, and Promptfoo are free, well-supported, and wide enough to sweep thousands of prompt variations quickly — tooling reviews note Promptfoo was already in use at more than a quarter of the Fortune 500. These scanners maintain high coverage across your threat model, but they embed assumptions about what attacks look like, and CSET's evaluation of 23 red-teaming tools found zero that provided explicit infrastructure for human scoring. That gap is where your people come in.

Human testers should focus on the techniques that scanners struggle to replicate — the ones the NVIDIA AI Red Team found most effective across six months of agent assessments:

• Social engineering — claiming to be "debugging" or an "admin user" proved "remarkably effective"; one tested agent wrote and executed a reverse shell on request.
• Frog-boiling (crescendo) attacks — progressive manipulation over multiple turns that gradually convinces an agent to reveal secrets. Multi-turn jailbreaks succeed 97% of the time within five conversation turns, per Vectra AI.
• Misdirection through legitimate workflows — a malicious pip install git+https://… bypasses agent resistance because library installation looks routine.

NVIDIA's team also found that a simple file-write tool can yield remote code execution by writing to ~/.bashrc, hooks.json, MCP.json, or skills files — a reminder that agent skills and tool configurations are attack surfaces in their own right. SkillzBench is being built around exactly that problem: free security scanning, sandboxed benchmarking, and a free MCP connector so teams can verify skills before equipping an agent with them.

Document every finding reproducibly — sample prompts, responses, and tooling — so fixes can be validated on retest. As HiddenLayer's practitioners put it, proving something can be broken without a method to fix it wastes everyone's time.

Fix What You Find: Architectural Controls Beat Prompt Guardrails

Your red team found a hole. Now the hard question: do you patch it with words or with architecture? After six months of agent assessments, NVIDIA's AI Red Team gave an unambiguous answer — prompt-based defenses are "routinely subverted", while architectural controls survive adversarial pressure.

The distinction matters because attackers adapt. Roleplay attacks succeed 89.6% of the time against large language models, and multi-turn jailbreaks succeed 97% of the time within five conversation turns. A guardrail that depends on the model politely refusing is a guardrail an attacker can talk their way around. A control that removes the capability entirely cannot be sweet-talked.

NVIDIA's team observed four recurring failure modes across their assessments: no access control, arbitrary code execution tools, no egress controls, and plaintext secrets. Their remediation priority list, in rough order:

• Agent access control — authenticate and authorize every agent action, not just the user
• Sandboxed command execution — treat arbitrary code execution as the single highest-impact risk in an accessible agent
• Default-deny network egress — an agent that can't call out can't exfiltrate
• Secrets in a dedicated manager — never plaintext in configs or memory
• Validated package repositories and least-privilege tools, MCPs, and skills

That last item deserves special attention. NVIDIA demonstrated that a file-write tool alone can yield remote code execution — not by writing malicious scripts directly, but by writing to files the system itself executes, such as ~/.bashrc, hooks.json, MCP.json, or skills files. The agent never needs a shell; it just edits its own configuration.

This is why least-privilege matters for skills and MCPs specifically. An agent that can only see the tools you explicitly gave it has a far smaller blast radius when a prompt injection lands. Misdirection through legitimate workflows — like a malicious pip install git+https://… — bypasses agent resistance precisely because library installation looks routine; a validated package repository closes that path.

As HiddenLayer's practitioners put it, proving something can be broken without any method to fix it wastes everyone's time — every test should carry a recommendation, and that recommendation should be architectural. SkillzBench is building toward this same principle from the supply side: security-scanning and version-pinning for agent skills, plus per-agent loadouts so an agent only ever sees the tools you chose for it, through a free MCP connector. Verified inputs and constrained permissions together shrink the attack surface your red team has to defend.

Make It Repeatable: Document, Retest, and Vet Every Agent Skill

A red-team report that proves a vulnerability but leaves no reproducible path to fix it is, in the words of one practitioner guide, a waste of everyone's time. Reproducibility is what separates a useful engagement from a screenshot.

Every finding needs three things attached: the exact prompt that triggered the failure, the model's full response, and a concrete remediation. Without those, the engineering team cannot validate the bug, the security team cannot confirm the fix, and the next model update silently reopens the same hole.

A reproducible report typically includes:

• Threat-model context — which system, which user role, which tools were active
• Attack payload — the verbatim prompt, including any multi-turn escalation steps
• Observed response — raw model output showing the policy violation or data leak
• Severity and impact — rated against a framework such as OWASP Top 10 for LLMs or MITRE ATLAS
• Recommended fix — architectural where possible, since NVIDIA's red team found prompt-based defenses are routinely subverted

Retesting closes the loop. After a fix lands, the same prompt must be replayed against the patched system to confirm the vulnerability is gone and that no regression appeared elsewhere. A scanner run once at launch leaves every later model update untested, so the attack library needs ongoing maintenance rather than a one-time license.

For agent-specific work, reproducibility extends to the skill files the agent loaded during the test. NVIDIA's six-month assessment flagged that file-write tools alone can yield remote code execution by writing to ~/.bashrc, hooks.json, MCP.json, or skills files, which makes the provenance of every skill in the agent's loadout part of the test record.

That provenance problem is exactly what SkillzBench is built to address. Published research has found roughly a quarter of agent skills carry at least one vulnerability, and there is currently no independent way to tell the safe ones from the risky ones before an agent loads them. SkillzBench is building a free directory that crawls and deduplicates skills from GitHub, pins every version to a commit SHA, and runs security scanning plus quality judging on each one. It will also benchmark every skill in a sandbox with the same model, with and without the skill, and publish the resulting lift in percentage points separately from any rating. A free MCP connector will let users toggle skills and tools per agent so the agent only sees what was chosen.

Skills, proven. Join the waitlist at skillzbench.com to get early access when the directory, scanner, and MCP connector go live.