How to Use AI Red Teaming: A Practical Walkthrough

The AI Security Gap: Why Your Systems Are Unguarded

Most organizations running AI systems today face a paradox: the threats are escalating faster than the defenses. In 2025, only 22% of organizations conducted adversarial AI testing, while 71% report severe impact from AI-powered cyberthreats and 60% admit they remain unprepared to defend against them.

The gap is widening, not closing. HackerOne recorded a 210% rise in AI vulnerability reports and a 540% rise in valid prompt injection reports in 2025 alone, with over 1,100 customer programs now placing AI in scope. Meanwhile, deepfake attack prevalence climbed to 47%, up from 30% in 2023.

Why traditional security tools fall short

Firewalls, endpoint detection, and static scanners were built for known malware signatures and network intrusion patterns. They do not understand natural-language instructions, so a crafted sentence slipped into a chatbot's context window sails right past them. Prompt injection, data leakage through retrieval pipelines, and model misconfigurations are vulnerabilities that traditional security tools just weren't built to handle.

The attack surface is also expanding in ways most security programs are not tracking. Autonomous agents and MCP servers connect models to live tools, databases, and APIs, creating new attack paths that standard penetration tests never touch. Over 80% of Fortune 500 companies are deploying active AI agents, multiplying the number of entry points an adversary can probe.

Where the exposure concentrates

• Prompt injection — adversarial instructions hidden in user input, retrieved documents, or tool responses that hijack model behavior
• Data leakage — sensitive context exfiltrated through retrieval-augmented pipelines or multi-turn conversations
• Agent tool abuse — autonomous agents calling MCP-connected tools in unintended or destructive ways
• Model misconfigurations — overly permissive system prompts, missing guardrails, unscoped API access

This is why AI red teaming exists. Rather than waiting for an external researcher to publish your flaw, you attack your own systems first — systematically, with real adversarial techniques — and fix what you find before production. It converts a defensive posture into an offensive one on your own terms.

For teams building on agent skills and MCP servers, the same logic applies to your supply chain. SkillzBench is building security scanning and sandboxed benchmarking for agent skills so you can evaluate and equip only the ones you trust, before they ever reach your stack.

Threat Model First: The Framework That Makes Red Teaming Work

Before you download a single scanning tool, you need to answer an uncomfortable question: what exactly are you testing for? A December 2024 workshop of academia, government, and private-sector participants run by CSET at Georgetown concluded that there are currently no established best practices for AI red teaming — which means the teams that succeed are the ones that bring their own structure.

That structure starts with a threat model. According to CSET's analysis of red-teaming design, the threat model is the core design feature of an exercise that influences all other choices: it bounds the scope and defines what counts as harmful output. In practice, that means documenting three things before any testing begins:

• The system itself — model type, modality, plugins, and any agent or MCP connections that expand the attack surface.
• The deployment context — who uses it, what data it touches, and what business impact a failure would have at runtime.
• What "harmful" means for your use case — prompt injection, data leakage, jailbreaks, or agentic risks, stated explicitly rather than left to the tool's defaults.

Once the threat model is set, two well-known frameworks fill in the details. Practitioner guidance from HiddenLayer describes the division of labor neatly: OWASP's Top 10 for LLMs tells you what to test, while the GenAI Red Teaming Blueprint tells you how — a phased process covering planning, execution, and post-engagement follow-up. Objective definition is the most important step; a vague scope dilutes the entire exercise.

Your choice of tool then depends on how your threat model aligns with what the tool already assumes. CSET's evaluation of 23 red-teaming resources found that tools embed built-in threat models, creating a trade-off: benchmark-style tools like NVIDIA's Garak — which tests roughly 100 attack vectors using up to 20,000 prompts per run — offer convenience but fix your threat model for you. Harness-style tools like the UK AI Safety Institute's Inspect offer deep customizability but require coding skill. Pick the one that matches your team's capability and your defined scope, not the other way around.

One caveat deserves emphasis: red teaming measures security, it does not assure it. As CSET's workshop participants put it, red teaming can prove a weakness exists but cannot guarantee one doesn't — treat results as a snapshot of outcomes under specific conditions, not a certificate of safety. That's why findings should always pair with remediation paths and re-testing after every model or system change.

This threat-model-first mindset extends to the components you attach to your agents. Because agent skills and MCP servers widen the attack surface, it helps to evaluate them with the same rigor — which is why SkillzBench is building security scanning and sandboxed benchmarking for agent skills, so provenance and risk are visible before a skill ever reaches your stack.

Your Free Testing Stack: Automated Tools Plus Human Ingenuity

You don't need a six-figure security contract to start breaking your own AI systems. The core red-teaming toolkit is free and open source — what you need is the judgment to aim it well.

Microsoft's PyRIT handles multi-turn attack orchestration, letting an attacking LLM refine prompts across a conversation rather than firing one-shot probes. NVIDIA's Garak brings breadth: according to tooling reviews, it tests roughly 100 attack vectors using up to 20,000 prompts per run across 20+ AI platforms. Promptfoo covers the newest ground — agent tracing and MCP testing — while CyberArk's FuzzyAI applies fuzzing and jailbreak detection to find crashes and bypasses.

Each tool earns its place in a practical stack:

• PyRIT — multi-turn orchestration when you need adaptive, conversational attacks
• Garak — broad automated sweeps across ~100 attack vectors
• Promptfoo — agent tracing, MCP testing, and compliance mapping to OWASP, NIST, MITRE ATLAS, and the EU AI Act
• FuzzyAI — fuzzing and jailbreak detection for edge-case failures

Automation has real limits, though. PyRIT's attacking-LLM refinement can take longer than attacking directly, and as practitioners note, adversaries continually adapt — which demands human ingenuity. Microsoft's own AI Red Team, after testing more than 100 generative AI products, concluded that human expertise stays essential even with automation in place.

One warning deserves special attention. A CSET evaluation of 23 red-teaming resources found that over half rely on AI-powered scorers — and none provided human-scoring infrastructure or documented reliability guarantees for those AI scores. As CSET puts it, AI scorers have "a significant impact on the validity of the evaluation results." Treat automated verdicts as leads for human review, not findings.

For teams building agentic systems, remember that skills and MCP servers expand the attack surface faster than most testing programs keep up. SkillzBench is building toward this gap — sandboxed benchmarking, security scanning, and a free MCP connector for evaluating agent skills — as a practical complement to red teaming your agent stack. Run the scanners for coverage, then put humans on the creative attacks no library contains yet.

Running the Exercise: Test Systems, Document Everything, Re-test Forever

The most common red teaming mistake is also the easiest to fix: testing the model in isolation instead of the deployed system. Real harms emerge from system interactions — a plugin, a retrieval pipeline, or an agent with tool access can turn a harmless model into a leaky one. A CSET workshop found that most exercises still test models alone, missing exactly these deployment-context risks.

Once testing starts, document everything. Share the sample prompts, the responses they triggered, and the tooling used so results are reproducible and fixes can be validated. A finding you cannot replay is a finding your engineers cannot fix or verify. Every documented weakness should also arrive with a remediation path — as HiddenLayer's practitioners put it, proving something can be broken without a way to fix it wastes everyone's time.

Structure your findings so they answer a simple question: did the guardrails actually work?

• Record the exact attack prompt, the raw response, and the configuration that produced it.
• Compare guarded versus unguarded versions of the system to isolate what your defenses actually block.
• Pair every finding with a specific fix — a filter change, a system-prompt revision, a permission restriction — and re-test to confirm it.
• Re-run the full suite after every model update, since results are a snapshot, not a certificate.

That last point matters more than most teams realize. Red teaming is point-in-time: outcomes shift with every model update, so it needs to live inside ongoing test-and-evaluation procedures rather than a one-time launch checklist. As one vendor bluntly advises, budget for upkeep of the attack library, not the license — a scanner run once at launch leaves every later model update untested.

For some organizations, this is no longer optional. Article 55 of the EU AI Act, effective since August 2025, converts adversarial testing into an auditable legal obligation for providers of general-purpose AI models with systemic risk, per market analysis. Even outside that scope, documented, repeatable testing is becoming the baseline expectation for enterprise AI procurement.

Agent stacks deserve special attention here. The shift to autonomous agents and MCP servers adds attack paths traditional testing doesn't cover, and agent skills themselves are part of the supply chain — published research found roughly 26% of agent skills carry at least one vulnerability. That's why SkillzBench is building pre-launch security scanning and sandboxed benchmarking for agent skills: every skill pinned to a verified commit SHA and tested in a sandbox before it ever reaches your agent. Red team your whole system, and vet the components you plug into it — the two practices work best together.

Building a Sustainable Red Teaming Practice

The most expensive mistake in AI red teaming isn't buying the wrong tool — it's running the right tool once and calling it done. Security practitioners consistently warn that a scanner run at launch leaves every later model update untested, so your budget should go toward maintaining the attack library, not the license itself (tooling analysis).

This matters because AI models change constantly. A red-teaming exercise is point-in-time: outcomes shift with every model update, deployment change, or new integration. CSET's December 2024 workshop — bringing together academia, government, and private-sector experts — concluded bluntly that no established best practices exist yet for AI red teaming, leaving teams to navigate "confusion and ambiguity" (CSET's analysis).

That gap has a practical consequence: treat every result as a snapshot, not a verdict. As CSET notes, red teaming can prove a vulnerability exists, but it can never guarantee one doesn't — results are simply "a snapshot of possible outcomes under specific conditions." The discipline measures security; it doesn't assure it.

Building a sustainable practice therefore means baking red teaming into ongoing test-and-evaluation, not treating it as a launch checkbox. A workable cadence looks like:

• Re-run your attack suite after every model update, prompt change, or new tool integration
• Keep the threat model current — agent stacks and MCP servers add attack paths faster than chat assistants ever did
• Pair every finding with a remediation path; proving something breaks without a fix wastes everyone's time
• Document prompts, responses, and tooling so results stay reproducible as your team grows
• Watch for new attack patterns — HackerOne logged a 210% rise in AI vulnerability reports and a 540% jump in valid prompt injections in 2025 alone (market research)

The stakes are also rising on the compliance side. Article 55 of the EU AI Act, effective August 2025, makes adversarial testing an auditable obligation for providers of general-purpose AI models with systemic risk. Yet only 22% of organizations conducted any adversarial AI testing in 2025, while 71% reported severe impact from AI-powered threats — a gap that regulation will increasingly force closed (adoption research).

If you're building agent stacks specifically, the skills and MCP servers you bolt on are part of that expanding attack surface. SkillzBench is building a free, independent answer: every skill version pinned to a commit SHA, security-scanned, and benchmarked in a sandbox, with a free MCP connector so your agents only see what you've deliberately equipped. Join the SkillzBench waitlist at skillzbench.com to get early access to verified, security-scanned skills for your agents.