AI Safety Research: What It Does, What It Costs, and When It's Worth It

The Safety Gap Nobody Budgeted For

In 2024, 78% of organizations reported using AI, up from 55% the year before. Yet as adoption accelerates, the discipline meant to keep AI safe is lagging badly behind — and nowhere is that gap more dangerous than in agentic systems with tool access.

The numbers tell a story of imbalance. The AI trust, risk and security management market is projected to grow from USD 2.8 billion in 2025 to USD 14.1 billion by 2033, according to market research. Meanwhile, Stanford HAI's AI Index reports that AI-related incidents are rising sharply while standardized responsible AI evaluations remain rare among major model developers. Money is flowing into AI faster than anyone is building the guardrails.

Agentic AI widens the gap in ways traditional security was never designed to catch. Academic researchers note that agent-specific vulnerabilities — session management flaws, model pollution, privacy leaks — are "not addressed by the frameworks used to build the agents, nor by research aimed at improving the agents," per an arXiv study. CrowdStrike puts the core problem bluntly: in traditional applications, security boundaries are defined by code and types; in AI agents, the security boundary is written in natural language. The reasoning chain itself becomes the attack surface, enabling threats like target hijacking, tool abuse, indirect prompt injection, and memory poisoning that conventional tools are structurally blind to.

The supply chain around agents adds another layer of exposure. Agent skills — pre-baked instruction packages — are a case in point:

• A proof-of-concept showed a malicious skill exfiltrating an entire codebase in just 4 user interactions, with zero log noise.
• Skills are "pre-baked behaviors," so a single one-click install can quietly wire exfiltration into your delivery flow.
• Reputation signals like stars and upvotes on public catalogs can be manipulated with bots and fake downloads.
• Without version pinning and change detection, attacks can persist undetected for extended periods.

This is exactly why SkillzBench is building a free directory that pins every skill to a commit SHA, runs security scans, and benchmarks each one in a sandbox before you connect it to your agent.

The encouraging news is that standardization is catching up. An IETF draft benchmark proposes 4 evaluation dimensions and 55 metrics for agent security, and new benchmarks like HELM Safety, AIR-Bench, and FACTS are emerging. But frameworks only matter if organizations adopt them — and today, most are scaling AI faster than they're securing it.

What AI Safety Research Actually Does

When an AI agent installs a new "skill," it's trusting a package of pre-written instructions that no traditional security tool was designed to inspect. That trust gap is exactly where modern AI safety research does its most concrete work.

The first function is evaluation before deployment. AI Trust, Risk and Security Management (TRiSM) solutions help organizations monitor, evaluate, interpret, and provide insights on machine learning models before they go live — a market worth USD 2.8 billion in 2025 and projected to reach USD 14.1 billion by 2033, according to Grand View Research. Yet standardized responsible-AI evaluations remain rare among major model developers even as AI incidents rise sharply, per the Stanford AI Index.

The second function targets agent-specific attacks. CrowdStrike researchers document threat classes that traditional tools are structurally blind to:

• Tool poisoning, where a trusted tool's description hides malicious instructions
• Tool shadowing, where a rogue tool rewrites how other tools appear to the agent
• Rugpull attacks, where an MCP server changes behavior after integration and, without version pinning and change detection, can persist undetected for extended periods

The stakes are vividly demonstrated in a proof of concept by Mitiga, in which a single malicious skill exfiltrated an entire codebase in just 4 user interactions — with zero log noise. "Skills aren't merely prompts," the researchers warn. "They're pre-baked behaviors, so a single one-click install can quietly wire exfiltration straight into your delivery flow." Reputation signals like stars and upvotes can be manipulated, so popularity is not a safety signal.

The third function is supply-chain vetting: checking the skills, MCP servers, and tools that agents connect to. This is the gap SkillzBench is being built to close — a free directory that pins every skill to a commit SHA, runs security scans and license checks, and benchmarks each skill in a sandbox so users can see its measured lift, with a free MCP connector that lets agents see only the skills you've chosen.

The encouraging sign is that this work is becoming quantitative. An IETF draft benchmark defines 4 evaluation dimensions and 55 second-level metrics — all expressed as pass rates — covering agent lifecycle risks from perception and memory to decision-making and execution. New safety benchmarks like HELM Safety, AIR-Bench, and FACTS are emerging alongside it. Safety research is turning from intuition into measurement, which is precisely what makes it possible to decide when it's worth the cost.

What It Costs — and What Actually Drives the Cost

Here's an uncomfortable truth about budgeting for AI safety: nobody publishes a price list. Direct dollar figures for safety research are scarce in public sources, so the honest way to answer "what does it cost?" is to look at where the money is actually flowing — and where it's getting stuck.

The clearest signal is market growth. The AI Trust, Risk and Security Management (TRiSM) market was valued at USD 2.8 billion in 2025 and is projected to reach USD 14.1 billion by 2033, a 22.6% compound annual growth rate, according to a Grand View Research market report. That trajectory tells you organizations are already spending heavily on monitoring, evaluation, and governance tooling — North America alone accounted for 31.3% of 2025 revenue.

But the raw compute behind safety evaluation is getting cheaper, not more expensive. Stanford's 2025 AI Index found that inference costs for GPT-3.5-level systems dropped more than 280-fold between November 2022 and October 2024, while hardware costs declined roughly 30% annually. Running an evaluation loop repeatedly is no longer the financial barrier it once was.

So what's the actual constraint? Two things, and neither is a software license:

• A shortage of skilled AI governance and cybersecurity professionals — cited in market research as a leading barrier to adoption, and one no tool purchase solves on its own.
• Operational cost concerns — the other top barrier, reflecting the ongoing burden of running evaluations, reviews, and monitoring as part of daily workflows rather than one-off projects.
• Process debt — version pinning, change detection, and sandbox testing of third-party components take sustained engineering discipline, and vendor research warns that attacks "can persist undetected for extended periods" without them.

That last point matters more than it sounds. When CrowdStrike researchers describe agent attacks that traditional security tools were never designed to detect, the implied cost isn't a bigger budget — it's new expertise and new process layered across security, engineering, and governance teams. The same market report that projects 22.6% annual growth also notes a persistent gap between companies recognizing AI risks and acting on them.

The practical reframe: stop asking "what does a tool cost?" and start asking "what does expertise and process cost?" A free, security-scanned skills directory with version pinning and benchmarking — like the one SkillzBench is building — addresses part of that process burden at zero tooling cost, but someone on your team still has to own the decisions.

Budget accordingly: the expensive part isn't the evaluation run. It's the people and workflows behind it.

ctaText: Skills, proven. Browse and test free, security-scanned agent skills on SkillzBench.

socialProofText: Published research found ~26% of agent skills carry at least one vulnerability — and one proof-of-concept exfiltrated an entire codebase in just 4 user interactions with zero log noise.

When It's Worth It: Stakes-Based Prioritization

Not every AI deployment deserves the same safety budget — and pretending otherwise wastes money on low-risk experiments while leaving genuinely dangerous systems under-defended. The smarter question isn't "should we invest in AI safety?" but "what happens if this specific system goes wrong?"

The evidence points clearly at where the stakes justify serious investment. Healthcare is the fastest-growing segment of the AI trust and security market at a 25.1% CAGR, alongside finance and autonomous driving — domains where, as market researchers note, "decisions impact human lives and well-being." The 223 FDA-approved AI medical devices recorded in 2023 (up from 6 in 2015) illustrate how quickly AI is moving into clinical contexts.

Agentic systems with tool access deserve their own tier of caution. Traditional security tools are structurally blind to reasoning-layer attacks like tool poisoning and tool shadowing because, as CrowdStrike's research explains, "in AI agents, the security boundary is written in natural language." A proof-of-concept from Mitiga showed a malicious agent skill exfiltrating an entire codebase with just 4 user interactions and zero log noise.

A practical prioritization framework looks like this:

• High stakes, invest heavily: healthcare, finance, autonomous driving, and any agentic system with tool or data access — full evaluation, governance frameworks like NIST AI RMF, and continuous monitoring.
• Medium stakes, invest selectively: customer-facing AI and workflows touching sensitive data — standardized benchmarking before deployment plus ongoing change tracking.
• Low stakes, run light: internal experiments and drafts — basic review and sandboxing may be enough.

One rule applies regardless of stakes: if third-party tools, skills, or MCP servers are in the loop, version pinning and change detection are your minimum viable safety practice. CrowdStrike documents rugpull attacks — where a tool behaves one way during evaluation and another after integration — and warns that "without version pinning and change detection, these attacks can persist undetected for extended periods." Reputation signals won't save you either, since catalog stars and upvotes can be manipulated with bots.

This is why SkillzBench is building its free directory around commit-SHA version pinning, security scanning, and upstream change tracking — so every skill your agent loads is pinned, scanned, and monitored, not just trusted on install. If your agents run on third-party skills today, that baseline discipline is worth it at any stake level.

Putting It Into Practice (Without a Governance Team)

Most teams adopting AI agents have no governance department — and they don't need one to start applying the lessons of agent security research. What they need is a short, disciplined set of habits that turns every skill and tool their agent touches from a blind spot into a vetted component.

The first habit is a mindset shift: treat every skill as an untrusted supply-chain component, not a harmless prompt. Security researchers at Mitiga demonstrated that a malicious agent skill can exfiltrate an entire codebase in just four user interactions, leaving no log noise behind. As they put it, skills are "pre-baked behaviors, so a single one-click install can quietly wire exfiltration straight into your delivery flow." Read a skill's instructions before installing it, the same way you'd review a dependency before adding it to production code.

Second, ignore the popularity signals. Stars, upvotes, and download counts are not safety signals — Mitiga's research shows reputation metrics on public catalogs can be manipulated with bot upvotes and fake downloads. A skill with tens of thousands of downloads can still carry a hidden payload.

Third, pin versions and watch for upstream changes. CrowdStrike documents "rugpull" attacks where an MCP server behaves innocently during evaluation, then changes behavior after integration, warning that "without version pinning and change detection, these attacks can persist undetected for extended periods." Pinning every skill to a specific commit SHA and re-checking when the upstream repo changes is the practical defense.

Fourth, swap ad-hoc testing for quantitative evaluation. The field is converging on structured measurement — an IETF draft benchmark defines 4 evaluation dimensions and 55 second-level metrics for agent security, all expressed as pass rates. You don't need all 55 metrics, but the principle scales down: test each skill with the same model, with and without the skill installed, and compare the difference. That's how you know whether a skill actually helps or just adds risk.

This is exactly the gap SkillzBench is being built to close. It's a free, independent directory of agent skills where every version is pinned to a commit SHA, security-scanned, license-checked, and benchmarked in a sandbox — with the performance lift shown separately from trust ratings, and money never touching scores. A free MCP connector will let you toggle exactly which skills each of your agents can see, and upstream changes trigger re-scans and alerts. It's pre-launch today; the waitlist is open if you want the vetted-skills layer ready when your agents are.

None of this requires a governance team. It requires treating the supply chain around your agent with the same suspicion you'd apply to any third-party code — because that's what it is.

• Review a skill's instructions before you install it — always.
• Never trust stars or download counts as a safety signal.
• Pin versions to a commit SHA and monitor upstream changes.
• Test skills with and without them installed, and compare.

Skills, proven. When SkillzBench launches, you'll be able to browse and test free, security-scanned agent skills in one place — join the waitlist to get first access.