What are the biggest challenges of using AI?

The Real Challenges of AI Adoption: Security, Trust, and Runaway Agents

If your AI agent can read your email, update your CRM, and run code, then every prompt injection or misbehaving skill becomes a direct threat to your systems — not just a bad chatbot answer. That is the uncomfortable reality of AI adoption in 2025: agents act unpredictably, and in doing so they expand your attack surface in ways traditional security was never designed to handle.

The numbers back this up. According to security research from Obsidian, 90% of AI agents hold excessive privileges across enterprise environments, and thousands of agents are deployed every week without any IT or security oversight. As one Fortune 500 security architect put it, the biggest risk is not what agents are designed to do — it is what they are allowed to do when compromised.

This is why agent risk differs fundamentally from LLM risk. Large language model threats mostly concern alignment: the model says something wrong or harmful. Agents, by contrast, "generate and execute actions via tools," meaning they can threaten real computing systems, applications, and resources. Academic research confirms that these agent vulnerabilities "are not addressed by the frameworks used to build the agents" — the tooling you use to build an agent does not automatically secure it.

The industry is responding. OWASP launched a dedicated GenAI Security Project, now backed by more than 200 supporting organizations and a community of over 30,000 members. Its Agent Control Standard makes the stakes plain: "Widescale adoption of AI agents depends on trust, and trust requires transparency and control." Its Top 10 for Agentic Applications even flags memory and context poisoning, noting that memory "is a feature" but "also an attack surface."

So what should a team actually watch for when adopting agents and the skills that extend them?

• Excessive privileges: agents with broad, standing access to systems and data they rarely need.
• Unvetted skills: instruction packages that may carry prompt injection, data exfiltration paths, or hidden assumptions about your environment.
• No runtime visibility: security teams often know only what an agent's configuration claims, not what it is actually doing.
• Silent drift: skills and agents change upstream, and regressions surface only after users report them.

The practical takeaway: treat agent capabilities and skills like any other software supply chain — verify provenance, scan for injection and exfiltration risks, and grant only the access a task requires. That is the gap SkillzBench is building to close, with security scanning, version pinning, and sandbox benchmarking for every skill it lists — all free. If you are choosing where agents fit in your workflow, start with the use cases where you can verify exactly what the agent sees and what it is allowed to do.

Flying Blind: Why Teams Without Evals Get Stuck in Reactive Loops

Ask a team how they know their AI agent is working, and you'll often get a shrug: "It seemed fine when we tried it." That instinct is exactly the problem. According to Anthropic's engineering team, the same capabilities that make AI agents useful—autonomy, intelligence, and flexibility—also make them harder to evaluate.

The most common substitute for real evaluation is what practitioners bluntly call vibes-based testing: running a few prompts and checking whether responses "look good." The MLflow team notes this approach is difficult to reproduce, doesn't scale, and can miss subtle regressions. OpenAI's guidance pushes in the same direction: instead of gut feel, teams need concrete eval questions—did the agent invoke the skill, did it run the expected commands—supported by a small structured set of 10–20 prompts, including negative-control cases where the skill should not trigger at all.

Why does this matter so much operationally? Because agents act across many turns and tool calls, mistakes propagate and compound. A small misfire on turn one becomes a wrong file edit, a bad query, or a misdirected email by turn five. Anthropic's warning is that teams without evals get stuck in reactive loops—catching issues only in production, where fixing one failure creates others. You patch the symptom, the patch breaks something else, and the cycle repeats.

Silent drift makes it worse. Skills and agent behavior change over time as models, prompts, and upstream dependencies shift, and regressions "often go unnoticed until users report them," as MLflow puts it. There's no alarm bell for a skill that quietly stops triggering or starts loading when it isn't relevant—one NVIDIA finding shows a skill that loads out of context can actually reduce agent performance.

The practical fix is to treat evaluation as an ongoing discipline, not a launch checkbox:

• Define success criteria before deployment—outcome, process, style, and efficiency—so two engineers aren't interpreting the same spec differently.
• Build a small structured eval set (10–20 prompts) with explicit, implicit, contextual, and negative-control cases.
• Re-run evals whenever a skill, model, or prompt changes, so regressions surface early rather than in user complaints.
• Use fast, targeted signals that catch specific regressions, rather than a single pass/fail verdict at the end.

This is also where use case matching earns its keep. NVIDIA's research found that per-product Skill Lift ranges from roughly +2 to +46 points while the harness itself differs by only about 5—meaning the domain, the task, and the evaluation design matter more than the tooling. And benefits aren't guaranteed: SkillsBench found curated skills improved average pass rates by 16.2 percentage points, yet 16 of 84 tasks actually got worse with a skill added.

That's the gap SkillzBench is building to close: sandbox-benchmarking every agent skill with and without it, publishing the Lift in percentage points, and re-scanning as upstream changes land. If you're choosing skills for a specific workflow, check verified, benchmarked evidence for your use case—browse the free directory at skillzbench.com and join the waitlist to test skills the moment they launch.

Use Case Mismatch: The Wrong AI Addition Can Actively Hurt Performance

The most counterintuitive finding in AI adoption research is that adding capability can make an agent worse. Skills and augmentations aren't uniformly helpful — in some cases they actively degrade performance, waste resources, and introduce new failure modes.

The evidence comes from two large-scale evaluations. A SkillsBench study found that curated skills raise average pass rates by 16.2 percentage points overall, but the effects range from just +4.5pp in Software Engineering to +51.9pp in Healthcare — and 16 of 84 tasks showed negative deltas, meaning the skills made those tasks harder. NVIDIA's SkillEvaluator research reached a similar conclusion after testing 300+ verified skills: "the domain, the task, and the evaluation design mattered more than the harness."

That last point deserves emphasis. The choice of agent framework (Claude Code vs. Codex) differed by only ~5 points on average, while per-product Skill Lift ranged from roughly +2 to +46. If you're agonizing over which agent to use while ignoring which skill fits your actual workflow, you're optimizing the wrong variable.

The costs of a mismatch aren't always obvious in quality scores, either. Consider these two NVIDIA examples:

• One skill (jetson-optimize-memory) reduced token usage by 76.9% and runtime by 53.7% — a clear win.
• Another (cuopt-install) *increased* tokens by 120.3% and time by 20.8% — the agent burned more resources to accomplish the same task.
• Across the SkillsBench tasks, nearly one in five got worse with skills added, not better.

Why does this happen? NVIDIA's researchers note that "every skill in an environment competes for the agent's attention, and a skill that loads when it is not relevant can reduce agent performance." A skill adds instructions, context, and decision points — all of which can distract the agent from the task at hand if they don't genuinely apply.

There's also a warning here for teams hoping AI can bootstrap itself. The SkillsBench researchers found that self-generated skills provide no average benefit — "models cannot reliably author the procedural knowledge they benefit from consuming." You can't just ask your agent to write its own skill and expect improvement.

The practical takeaway: treat every skill addition as a hypothesis to test, not a default upgrade. Run a small eval set — OpenAI recommends 10–20 prompts including negative-control cases where the skill shouldn't trigger — and compare performance with and without the addition before committing.

This is precisely the gap SkillzBench is building to close: benchmarking every listed skill in a sandbox with the same model, with and without the skill, and publishing the Lift in percentage points so you can see whether an augmentation actually helps your kind of task before you adopt it. Every benchmark and the directory will be free on skillzbench.com — join the waitlist to get access the moment it launches.

The Skill Supply Chain: Hidden Risks in What Your Agent Loads

Your AI agent is only as trustworthy as the instructions it loads — and most teams have no idea where those instructions came from. As agent skills become a standard way to extend Claude Code, Codex, Cursor, and Gemini CLI, they introduce a supply-chain risk that traditional software vetting was never designed to catch.

The first failure mode is deceptively mundane. A skill's name and description are the primary signals an agent uses to decide when to invoke it, and OpenAI's engineering guidance warns that vague or overloaded descriptions mean "the skill won't trigger reliably." OpenAI groups the deeper problem into three hidden assumptions: triggering assumptions (skills that never fire, or fire unintentionally), environment assumptions (a skill that assumes npm is installed on the user's machine), and execution assumptions (a skill that skips steps the author considered obvious).

Then comes the security layer. Unlike a chatbot, an agent "can threaten real computing systems, applications, and resources because it generates and executes actions via tools," as academic research on agent vulnerabilities puts it — and those vulnerabilities are not addressed by the frameworks used to build the agents. NVIDIA's skill evaluation pipeline treats scanning as a prerequisite for adoption, checking every skill for prompt injection, Unicode smuggling, data exfiltration, PII, and secret detection before installation. Published research has found roughly 26% of skills carry at least one vulnerability.

OWASP's agentic Top 10 formalizes the threat landscape:

• Prompt injection and Unicode smuggling that hide malicious instructions inside skill content
• Data exfiltration paths that leak sensitive context to external endpoints
• Memory and context poisoning — "Memory Is a Feature. It Is Also an Attack Surface," per OWASP's GenAI Security Project
• Model pollution, where individually benign prompts become harmful when concatenated as training data

The behavioral risks compound the security ones. A skill that loads when it is not relevant competes for the agent's attention and can actively degrade performance — one NVIDIA-tested skill increased token usage by 120.3% instead of saving resources. Verification cannot be assumed either: while NVIDIA reported a 97/100 security baseline for its verified skills, enterprise security analysis finds 90% of agents hold excessive privileges, with thousands deployed weekly without IT oversight.

This is why provenance matters as much as performance. SkillzBench is being built to close exactly this gap: every skill version pinned to a commit SHA, security-scanned and license-checked, then benchmarked in a sandbox before it reaches your agent — with Lift always shown separately from trust scores. Until independent verification is the default, treat every skill your agent loads like an unvetted dependency, because that is precisely what it is.

A Practical Playbook: Match, Evaluate, Verify, and Stay Current

Knowing the challenges is one thing; acting on them is another. The research points to a surprisingly compact playbook — five steps that turn "AI might help us" into a decision you can defend.

Step 1: Match to the use case, not the framework. When NVIDIA evaluated 300+ verified skills across two agent harnesses, the harness difference averaged only ~5 points, while per-product Skill Lift ranged from +2 to +46 — the domain, task, and evaluation design mattered more than the harness. And benefits swing wildly by domain: curated skills lifted pass rates anywhere from +4.5 points in software engineering to +51.9 points in healthcare.

Step 2: Define success criteria before deployment. Anthropic's engineering team notes that ambiguity in success criteria is a documented failure mode — two engineers can interpret the same spec differently, and eval suites resolve this by explicitly encoding expected behavior. Write down what "good" looks like (outcome, process, style, efficiency) before anything ships.

Step 3: Run a small, structured eval set. You don't need a research lab — OpenAI recommends 10–20 prompts per skill, mixing explicit, implicit, contextual, and negative-control cases where the skill should not trigger. This matters because additions can backfire: one NVIDIA skill increased token usage by 120.3%, and 16 of 84 SkillsBench tasks got worse with curated skills.

Step 4: Security-scan and version-pin every skill. Agents "can threaten real computing systems, applications, and resources because they generate and execute actions via tools," and academic research confirms these vulnerabilities aren't addressed by the frameworks themselves. Treat skills like any software dependency:

• Pin to a specific commit SHA, not "latest"
• Scan for prompt injection, data exfiltration, and secret detection before install
• Check licenses before the skill touches your codebase
• Benchmark with and without the skill to measure actual Lift

Step 5: Stay current. Skill regressions "often go unnoticed until users report them," per the MLflow team — so track upstream changes and re-test after every update.

This is exactly the workflow SkillzBench is building to make routine: every listed skill version-pinned to a commit SHA, security-scanned and license-checked, then sandbox-benchmarked with the same model with and without the skill — with Lift always shown separately from trust scores. Change alerts re-scan and re-benchmark when upstream code moves, and everything, including the MCP connector, stays free.

Ready to adopt skills with evidence behind them? Join the free SkillzBench waitlist and get verified, benchmarked agent skills the moment they launch — no cost, no pay-to-rank, ever.