How to learn AI red teaming?

Why AI Red Teaming Skills Are in Demand (and Why It's Hard to Learn)

If you want a cybersecurity skill where demand is exploding faster than the talent pool, AI red teaming is it — but the catch is that almost nobody can tell you the "right" way to learn it.

The numbers tell a striking story. According to market research, the AI red teaming services market is projected to grow from $1.75 billion in 2025 to $6.17 billion by 2030 — a compound annual growth rate of roughly 28.5%. The same report names Google, Microsoft, Deloitte, IBM, and CrowdStrike among the organizations actively hiring in this space.

On the demand side, the pressure is real. Aggregated industry statistics show that 97% of enterprises have already encountered generative AI-related security incidents, yet only 26% conduct proactive AI-specific security testing. That gap between threat exposure and testing readiness is precisely where skilled red teamers become valuable.

Here's the problem: you can't just enroll in a standard course. Experts at Georgetown's CSET note the field is "severely lacking in theoretical grounding and technical best practices," with workshop participants agreeing that no established practices exist — creating genuine confusion for newcomers. Even the tooling landscape has "no standard stack or dominant platform," according to Palo Alto Networks.

Part of why learning is hard: AI red teaming is fundamentally different from traditional red teaming. It's not a skill you can transfer one-to-one from conventional security work:

• Different target — traditional red teaming attacks infrastructure; AI red teaming attacks model behavior, including hallucinations, misuse, and prompt manipulation.
• Different logic — AI systems are probabilistic, not deterministic, so the same attack can fail once and succeed the next time.
• Different team shape — effective AI red teams combine ML experts, security professionals, and social scientists rather than pure penetration testers.
• Different lifecycle — testing is iterative and continuous, evolving alongside the model rather than ending at a fixed engagement date.

The stakes sharpen the picture further: multi-turn jailbreaks reach roughly 97% success within five conversational turns, and role-play attacks succeed nearly 90% of the time — techniques that require human creativity no scanner fully replicates. CSET researchers even caution that generative AI systems "struggle with creativity," the very skill effective red teaming demands.

For those building AI agents rather than breaking them, the same adversarial thinking applies to your supply chain — published research found roughly 26% of agent skills carry at least one vulnerability. SkillzBench is building free security scanning and benchmarking for exactly this problem, so you can verify skills before they ever reach your agent.

The takeaway: demand is massive, the field is young, and the absence of a standard curriculum is an opportunity for anyone willing to build their own structured learning path.

Start With Frameworks and Threat Modeling, Not Tools

It's tempting to jump straight into installing red-teaming tools, but the experts who study this field for a living say that's backwards. Before you touch Garak or PyRIT, you need a mental map of what you're defending against — and that map comes from frameworks and threat modeling, not software.

The clearest starting point is a pair of free resources from OWASP. As HiddenLayer's best-practices guide explains, the OWASP Top 10 for LLM Applications tells you what to test, while the OWASP GenAI Red Teaming Guide defines how to test it — the latter is explicitly described as "a great starting point for new red teamers." Together they give a beginner a structured vocabulary for vulnerabilities like prompt injection, which shows up in 70% of security audits as the #1 LLM weakness.

Once you know the vulnerability categories, the next skill is threat modeling. Researchers at Georgetown's CSET call it "the core design feature of a red-teaming exercise that influences all other design choices" — in other words, your threat model determines your team, your attack methods, and even how you interpret results. Skipping this step is why so many ad-hoc tests produce noise instead of insight.

It also helps to understand that "AI red teaming" actually covers three distinct disciplines:

• AI red teaming — broad and deep, creative and adaptive, probing model behavior for misuse and manipulation.
• AI penetration testing — narrow and deep, focused on known exploits with proof-of-concept attacks.
• AI security scanning — broad and shallow, automated pattern detection suited to CI/CD pipelines.

Knowing which discipline you're practicing keeps your expectations — and your results — honest.

For process, Palo Alto Networks lays out a six-step path worth memorizing: define objectives and scope with a specific scenario in mind; build a multidisciplinary team; choose your attack methods and tools; establish a safe, isolated testing environment (never production, and log everything including failed attempts); analyze results and prioritize remediation; then rerun, retest, and refine as you integrate testing into the development lifecycle.

This framework-first mindset applies well beyond your own testing exercises. Published research has found that roughly 26% of agent skills carry at least one vulnerability, which is why SkillzBench is building free security scanning and version-pinned verification into its skills directory — so the same "know the threat model before you trust the component" discipline applies to what you load into your agents. The frameworks cost nothing to learn; the discipline they build is what separates useful red teamers from tool collectors.

Master the Open-Source Toolchain: Garak, PyRIT and Promptfoo

Once you understand threat models, the fastest way to build real red-teaming skill is to run the open-source toolchain against a live model. The field has consolidated around three vendor-backed tools — NVIDIA's Garak, Microsoft's PyRIT, and Promptfoo — and each occupies a distinct stage of a four-stage pipeline.

Garak is your discovery engine. It ships with 80+ probes spanning jailbreaks, prompt injection, and data leakage, and version 0.13 added conversation support and ASCII smuggling probes, per technical tool evaluations. Point it at a test endpoint and let it map your attack surface broadly before you invest deeper effort.

PyRIT takes over where Garak stops: orchestrated, multi-turn campaigns. Microsoft has applied it across 100+ red-teaming operations on Azure AI products, and it matters because attack research shows multi-turn jailbreaks reach ~97% success within five conversational turns — exactly the scenario single-shot scanners miss.

Promptfoo then locks findings into CI/CD as regression gates. It covers 50+ vulnerability types including OWASP LLM01–LLM10 and agent API flaws like BOLA/BFLA, so every model update gets retested automatically rather than scanned once at launch.

For defensible numbers, layer in HarmBench (200 behaviors across 6 harm categories) and JailbreakBench (100 harmful plus 100 benign behaviors). The staged-pipeline guidance is blunt about the common failure mode: "The mistake most teams make is using one tool for all four stages. Garak finds broad surface; PyRIT runs targeted multi-turn campaigns; HarmBench produces defensible numbers. They are not substitutes."

A few practical warnings before you start:

• Never test against production — isolate model versions, enforce rate limits, and log failed attempts too.
• Budget for engineering time and API spend, not licenses — the open-source tools are free, and PyRIT's LLM-as-judge scoring adds 2–4× raw inference cost versus pattern matching.
• Keep the attack library maintained; a scanner run once at launch leaves every later model update untested.

The same verification logic applies to the agent layer, not just models. Published research found roughly 26% of agent skills carry at least one vulnerability — which is why SkillzBench is building security scanning and sandboxed benchmarking for skills before they reach your agents via its free MCP connector. Red-team your tools the way you red-team your models.

Build the Manual Skills Automation Can't Replace

Here's the uncomfortable truth about AI red teaming tools: the attacks that actually break models are conversational, creative, and stubbornly human. Automated scanners will hand you a baseline, but the vulnerabilities that matter most require a person who knows how to talk a model into failure.

The numbers back this up. Multi-turn jailbreaks reach roughly 97% success within just five conversational turns, and role-play attacks — where you convince a model to adopt a persona that bypasses its guardrails — succeed 89.6% of the time, making them the highest-risk jailbreak vector. These aren't one-shot exploits. They're patient, adaptive dialogues that no static probe can replicate.

Why does automation fall short? CSET researchers put it bluntly: generative AI systems struggle with creativity, which is a fundamental skill for effective red teaming. Automated tools also miss entire vulnerability classes — indirect prompt injection, multi-hop agent chains, RAG poisoning, and model inversion — that require understanding how a system's components interact, not just how a single prompt behaves.

So how do you build these manual skills? Practice the attack patterns with documented success rates:

• Role-play framing — assign the model a persona, then escalate requests through that character (89.6% success rate)
• Bad-math misdirection — bury harmful requests inside arithmetic or logic errors (76% success rate)
• Iterative reframing — rephrase a refused request across multiple turns until guardrails erode
• Contradiction exploitation — feed the model conflicting instructions and probe which constraint it drops (53% success rate)

Discipline matters as much as creativity. Palo Alto Networks' guidance is unambiguous: never test against production. Work in an isolated environment with pinned model versions, enforce rate limits, and log everything — including your failed attempts, which often reveal as much about a system's defenses as your successes.

Then close the loop. HiddenLayer's rule is worth tattooing on your workflow: every finding needs a remediation recommendation. Proving something can be broken without a method to fix it, they argue, wastes everyone's time. Share your sample prompts, responses, and tooling so results can be reproduced and fixes validated.

This same adversarial thinking applies beyond your own models. Published research suggests roughly 26% of agent skills carry at least one vulnerability — which is why SkillzBench is building free security scanning and sandbox benchmarking for agent skills, so you can verify what you're loading into an agent before it ever touches your workflow.

The tools get you coverage. The human gets you breakthroughs. Master both, and you'll be practicing the version of AI red teaming that automation still can't touch.

Apply Red-Team Thinking to Your Own AI Stack Today

You don't need a lab or a security budget to start red-teaming today — the AI skills and tools you already load into your agents are the perfect first target. Research on the agent-skills ecosystem found that roughly 26% of published agent skills carry at least one vulnerability, and prompt injection — the #1 LLM vulnerability — shows up in 70% of security audits.

Here's what that means in practice: every SKILL.md instruction package, MCP tool, or third-party skill you connect to Claude Code, Codex, Cursor, or Gemini CLI is an instruction source your agent will follow. If an attacker embeds a malicious instruction in a skill, your agent may execute it without ever flagging it to you. The multi-turn jailbreak techniques covered earlier in this article — role-play framing, bad-math misdirection, iterative reframing — work just as well against a skill's embedded instructions as against a chat model.

Apply the red-team mindset with a simple vetting pass before any skill reaches your agent:

• Read the instructions yourself. Open the SKILL.md and ask: does anything here tell my agent to fetch external content, send data somewhere, or override my system prompt?
• Check the supply chain. Who published it, when was it last updated, and does the license check out? Unmaintained skills accumulate risk as upstream dependencies change.
• Test in isolation first. Run the skill against a sandbox agent with no access to production data, and log every output — including the ones that look fine.
• Pin the version. Attach the skill to a specific commit SHA so an upstream change can't silently alter what your agent executes.

That last point matters more than most people realize. A skill that was safe at install time can become a liability after its maintainer pushes an update — the same reason security practitioners warn that a scanner run once at launch leaves every later model update untested.

Doing this manually for every skill is tedious, which is exactly the gap SkillzBench is building to close. Its platform — currently in development, with a waitlist open — will security-scan, license-check, and version-pin agent skills, and benchmark each one in a sandbox to measure whether it actually lifts agent performance. The free MCP connector will let you toggle exactly which vetted skills and tools each agent can see, so nothing unvetted slips into context. If you want first access when the scanning and benchmarking go live, joining the waitlist is the simplest step.

Red-teaming your own stack won't make it invulnerable — as Georgetown's CSET researchers put it, red-teaming measures security rather than improving it. But measuring it beats assuming it, and the skills you vet today are the incidents you avoid tomorrow.