AI Vulnerabilities: What It Does, What It Costs, and When It's Worth It

The Discovery Paradox: Why Finding More Vulnerabilities Hasn't Made Anyone Safer

Something strange is happening in security right now: the tools that find vulnerabilities have never been better, and yet nobody feels safer. That's not a coincidence — it's the central paradox of AI vulnerability tooling, and it should change how you evaluate any security purchase.

Detection capability has genuinely exploded. On the CyberGym academic benchmark, large language models went from finding under 20% of vulnerabilities at the start of the previous year to over 85% today, according to Anthropic's vulnerability research. OpenAI reports its GPT-5.5-Cyber model scoring 85.6% on the same benchmark — its highest single-model result. In raw finding power, AI has delivered.

But finding isn't protecting. OpenAI states it plainly: "vulnerability reports, on their own, do not protect anyone," and argues that the bottleneck has shifted from finding to patching. Defenders are now drowning in reports they lack the capacity to triage, let alone fix. Anthropic's own numbers show the imbalance: over 29,000 candidate vulnerabilities discovered in six months, with only around 6,000 manually reviewed. Every unreviewed report is detection capacity that produced zero security value.

The problem gets worse when the detection itself can be fooled. Security firm Trail of Bits demonstrated that a fake AI agent skill could slip past ClawHub's detector, Cisco's scanner, and all three skills.sh scanners — because of a structural blind spot known as time-of-check versus time-of-use. As The Hacker News reported, "a scanner checks a fixed package, while an attacker can keep tweaking the payload until it passes." Scanners analyze the submitted package, but skills often instruct agents to fetch content from external links — and that content can be swapped after the scan clears.

• Detection rates above 85% sound impressive, but unpatched findings protect nothing.
• Human review capacity — roughly 6,000 of 29,000+ findings at Anthropic — is the real constraint.
• One-time scans miss content that changes after the check, so trust signals like stars and clean scan results can be gamed.
• Volume without triage creates noise, and noise buries the vulnerabilities that actually matter.

This is why the smartest evaluation question isn't "how many vulnerabilities does this tool find?" It's "what does this tool do after it finds something?" Tools that produce validated, actionable findings with remediation guidance deliver value; tools that dump raw scan volume onto an already-overwhelmed team mostly add work. That principle shapes what SkillzBench is building for agent skills: continuous re-scanning and re-verification when upstream code changes, rather than a single point-in-time check that attackers can simply wait out.

The market is pouring money into detection — the AI vulnerability scanning segment alone is projected to grow from $2.41 billion in 2024 to $9.09 billion by 2034. The buyers who come out ahead will be the ones who spend that money on the full loop: find, verify, fix, and re-check.

What AI Vulnerability Tools Actually Do — And Where They Still Fail

AI vulnerability scanners have gotten genuinely good at finding bugs — and that's exactly why their blind spots matter more now. On the CyberGym benchmark, LLM detection rates jumped from under 20% to over 85% in a single year, and OpenAI's GPT-5.5-Cyber scored 85.6%, its highest single-model result on that benchmark.

The commercial and free tools deliver real results in practice. Anthropic's OSS Scanner, free for open-source projects, surfaced over 29,000 candidate vulnerabilities in six months, and independent pen testers confirmed 88% of critical/high-severity reports met a proper coordinated-disclosure bar, with only one false positive. Maintainer testimonials back the signal quality: wolfSSL found that of 74 reports, all but two were valid and five became CVEs; curl's Daniel Stenberg credited the scanner with finding "one of the worst curl vulnerabilities reported in the last few years"; and an OpenSSL maintainer said reports came with real exploits attached — "basically job done for an engineer." OpenAI's Codex Security has scanned 30+ million commits across 30,000+ codebases.

But capability has a ceiling, and it's structural. Security researchers at Trail of Bits bypassed ClawHub's detector, Cisco's scanner, and all three skills.sh scanners, concluding that "a scanner checks a fixed package, while an attacker can keep tweaking the payload until it passes." The core failure: scanners analyze the submitted package, not the external links a skill tells agents to fetch — and attackers can swap externally hosted payloads after the scan clears.

Trust signals fail alongside scanners. In the AIR fake-skill experiment, a malicious skill ("brand-landingpage") reportedly reached roughly 26,000 agents — including some on corporate accounts — from a marketplace repo with about 36,000 GitHub stars. As reporting on the incident put it, none of the signals people lean on caught it: not the scanners, not the stars, not the open-source reputation.

• Treat every agent skill as software with a full supply chain, not a trusted snippet.
• Pin each skill version to a specific commit so nothing changes silently.
• Re-verify whenever upstream content changes — a one-time scan is not protection.
• Vet external links a skill instructs agents to fetch, since payloads can live outside the package.
• Enforce least privilege so agents only see the skills you deliberately chose.

The takeaway for businesses adopting agents: discovery without re-verification creates a false sense of security, and published research suggests ~26% of agent skills carry at least one vulnerability. That's the gap SkillzBench is building against — pinning every skill version to a commit SHA, security-scanning, and continuously re-scanning and re-benchmarking when upstream changes occur, with a free MCP connector so your agent only sees vetted, chosen skills. Scanners find bugs; ongoing verification is what keeps agents safe.

What It Costs: Breach Math, Hidden Remediation Bills, and Free Alternatives

The average data breach now costs USD 4.88 million, but organizations using AI for vulnerability management report USD 3.76 million — a gap worth paying attention to before you sign any contract. Cost avoidance, not feature lists, is where the business case lives.

According to market research on AI vulnerability scanning, extensive AI security usage cuts breach costs by up to 33%, and IBM breach data shows AI and automation in security operations save roughly $2 million per breach. Skills shortages add about $180,000 to the average breach, while mean time to identify and contain a breach has climbed to 247 days — the first increase after a five-year decline.

That last number exposes the hidden bill. A scanner without remediation capacity delivers little value, because the bottleneck has shifted from finding vulnerabilities to patching them. OpenAI states plainly that "vulnerability reports, on their own, do not protect anyone," and Anthropic's own data shows only about 6,000 of more than 29,000 discovered vulnerabilities were manually reviewed. Budget for fix capacity, not just discovery.

Free options exist, though they come with limits:

• Anthropic's OSS Scanner is free for open-source projects and has surfaced real issues in curl and wolfSSL.
• OpenAI's Patch the Planet offers participants ChatGPT Pro, conditional Codex Security access, and API credits.
• Small and mid-sized enterprises increasingly use Security-as-a-Service to access enterprise-grade protection without significant upfront investment.

One honest gap: specific vendor pricing for commercial AI vulnerability tools is not publicly documented in the sources reviewed here, so treat any quote as negotiable and tied to scope.

Cost math also breaks down when a scan clears something that later turns malicious. Independent testing by Trail of Bits bypassed ClawHub's detector, Cisco's scanner, and all three skills.sh scanners, concluding that "a scanner checks a fixed package, while an attacker can keep tweaking the payload until it passes." That is precisely the gap SkillzBench is built to close — pinning every skill version to a commit SHA, security-scanning, and continuously re-scanning when upstream changes land, all free.

When It's Worth It: A Decision Framework for Your Business

A scanner that finds 500 vulnerabilities your team can fix is worth more than one that finds 5,000 it can't. That's the uncomfortable truth behind AI vulnerability tooling: the value isn't in discovery, it's in whether you can act on what gets discovered.

The strongest buy signal is breach-cost exposure. Organizations using AI for vulnerability management report breach costs of USD 3.76M versus the USD 4.88M average, and IBM data shows AI and automation in security operations save roughly $2 million per breach. If you operate in a regulated industry — GDPR, NIS2, HIPAA, or PCI-DSS territory — continuous monitoring is effectively table stakes, since these regulations are actively compelling investment in vulnerability management, per market analysis from Grand View Research.

AI vulnerability tooling likely pays off when:

• You face high breach-cost exposure or regulatory mandates that make a $3.76M-versus-$4.88M outcome material to the business.
• You have real remediation capacity — OpenAI puts it plainly: "vulnerability reports, on their own, do not protect anyone," and the bottleneck has shifted from finding flaws to patching them.
• Shadow AI is growing in your organization — unsanctioned employee AI use more than doubled to 43% of breaches, up from 20%.
• You maintain an asset inventory, because you can't scan and protect what you don't know exists.

Conversely, the purchase is premature if you lack fix capacity, asset visibility, or basic access controls — only 40% of AI-breach victims had access controls on their models and data. Buying discovery tooling without remediation capacity creates a false sense of security, not protection.

Wherever you land, secure the AI perimeter before the model itself. Compromised APIs, applications, and plug-ins accounted for 27% of AI-related breaches, and cloud misconfigurations another 27% — attackers are landing in the infrastructure around AI, not the model. As one security expert framed it, these attacks land in the telemetry of cloud and identity environments, so the defense is behavioral detection across everything the AI touches. Treat your models and their APIs like crown jewels.

Finally, get ahead of governance. NIST's Center for AI Standards and Innovation is actively soliciting input on securing AI agents, noting many companies adopted them without plans to mitigate their flaws. Organizations that govern unsanctioned AI use now will be better positioned for both cost control and whatever compliance requirements emerge. If your teams use agent skills, the same logic applies: treat skills as software, vet external links, pin versions, and re-verify on change — the approach SkillzBench is building into its free skills directory and MCP connector, so agents only see vetted, chosen tools.

Putting It Into Practice: Securing Your Agent Skills Without a Security Team

You don't need a dedicated security team to apply the core lessons from the scanner-bypass research. The defender guidance is straightforward, and each piece maps to something you can do — or automate — today.

The starting point is a mindset shift: treat agent skills like software, not like prompts. A skill is a package of instructions that can fetch external content, run tools, and change what your agent does. That means it deserves the same vetting you'd give any dependency — and the research shows why. Trail of Bits bypassed multiple production scanners because, as they put it, "a scanner checks a fixed package, while an attacker can keep tweaking the payload until it passes."

The recommended practices break down into four habits:

• Vet every external link a skill references — scanners only analyze the submitted package, not content the skill tells your agent to fetch later.
• Pin versions, so you know exactly what you installed and can detect silent changes.
• Re-check whenever upstream content changes, since payloads can be swapped after a scan clears.
• Enforce least privilege — only give each agent the skills and tools it actually needs.

The last point matters more than most teams realize. Breach data shows compromised APIs, plug-ins, and cloud misconfigurations were the top entry points for AI-related breaches at 27% each, and only 40% of AI-breach victims had access controls on their models and data. Locking down what your agents can see and touch is one of the highest-leverage moves available.

This is exactly the problem SkillzBench is being built to solve. Every skill version is pinned to a commit SHA, security-scanned, and continuously re-scanned and re-benchmarked when upstream changes occur — automating the re-checking habit that's nearly impossible to maintain by hand. And a free MCP connector means your agent only ever sees the skills you've explicitly chosen and vetted, putting least privilege into practice by default rather than by policy memo. Everything — the directory, the MCP, benchmarks, and alerts — will be free.

Two caveats are worth keeping in mind. First, no scan is permanent: the fake skill in the AIR experiment reportedly reached roughly 26,000 agents despite passing scanners, and none of the usual trust signals caught it — not the scanners, not the GitHub stars, not the open-source reputation. Second, published research found about 26% of skills carry at least one vulnerability, so treat vetting as ongoing hygiene, not a one-time gate.

If you want to stay ahead of this space as standards mature — NIST is actively developing AI agent security guidance — the SkillzBench newsletter and waitlist are open now. You'll get research-backed updates on agent skill security as the platform comes together, no payment or commitment required.