Can AI Do Data Analysis? Where It Helps and Where It Hallucinates

Why AI Is Already Your Analyst (and Why That's Risky)

If you've asked an AI assistant to summarize a spreadsheet or write a quick Python script this week, you're in good company — and you're also standing on a fault line.

The numbers show AI has quietly become the default analyst in exactly the fields that trade on analysis. Federal Reserve research finds adoption concentrated in "cognitive and analytical work": financial services shows 30% firm adoption with 63% of workers using generative AI, and professional services sits at 33% with 62% of workers doing the same. The OECD puts ICT-sector firm AI use even higher, at 57.3%.

And on the individual level, it's working. McKinsey's State of AI survey found that 80% of respondents say AI has improved their personal productivity, and 50% say it helps them make better decisions. In data analysis specifically, the tools are genuinely capable — practitioner testing confirms all three major assistants can generate Python or R code, run exploratory data analysis on uploaded files, and a peer-reviewed study found six tested assistants achieved 100% accuracy on statistical test selection.

Here's the fault line: those same tools can fabricate the numbers you're paying them to find. The same testing found all three assistants "can hallucinate statistics — meaning they sometimes generate numbers, citations, or study results that don't actually exist." IBM is blunt about why: generative AI "does not 'know' what is true or false. It only 'knows' what fits the pattern." Hallucinations can be reduced, but not eliminated.

The result is a widening trust gap between individual wins and enterprise results. Only 37% of organizations in McKinsey's survey report EBIT contribution from AI — essentially flat from the prior year — while nearly nine in ten report using AI somewhere in the business. The productivity is real; the reliable, decision-grade output is not automatic.

So where does the gap come from? Three documented patterns:

• Confident methodological errors — AI can recommend a statistically valid procedure while ignoring whether it fits your dataset, missing assumptions like independence or unequal variances.
• Reasoning failures on complex tasks — Stanford HAI's AI Index documents that models often fail logic tasks "even when provably correct solutions exist."
• Cascading hallucinations — in multi-agent pipelines, one fabricated metric can get treated as verified fact by every downstream agent, compounding the error.

That last one matters most as teams wire AI agents into analytical workflows. McKinsey found 40% of large organizations are now scaling AI agents, up from 27% — yet only about 20% of companies report mature frameworks for managing them. If you're connecting agents to your data stack, verifying what those agents (and their skills) actually do is the cheapest insurance available; SkillzBench is building exactly that kind of free, independent verification for agent skills, so you know what's trustworthy before it touches your data.

What AI Genuinely Does Well in Data Analysis

If you hand an AI assistant a messy CSV and ask it to find patterns, the odds are better than most skeptics assume. Practitioner testing of ChatGPT, Claude, and Gemini found that all three can generate working Python and R code, explain statistical concepts, and run uploaded files inside sandboxed code-execution environments — with ChatGPT's data analysis environment described as the most mature for exploratory data analysis.

The strongest evidence comes from a 2025 peer-reviewed study cited in that testing: all six AI assistants evaluated achieved 100% accuracy on statistical test selection across 20 scenarios. That means every model correctly identified which test fit each research question — a task that trips up plenty of human analysts. Two caveats matter, though: the study used free-tier models on a single day, so treat it as a promising snapshot rather than a guarantee.

Why does AI do so well here? Because these are narrow, well-defined tasks. Choosing a t-test versus a chi-square has clear rules; generating a pandas snippet has verifiable output. The same models that ace test selection can fail badly on open-ended, multi-step reasoning — the Stanford HAI AI Index documents that models often fail logic tasks even when provably correct solutions exist. Well-specified problems play to AI's strengths; ambiguity exposes its weaknesses.

Each major assistant also has a distinct flavor, which matters when you're picking one for analytical work:

• ChatGPT offers the most mature sandboxed environment for exploratory analysis on uploaded CSV and Excel files, though it sometimes pushes forward without flagging data issues.
• Claude produces the cleanest code for complex or multi-file tasks, but can bury the core answer under so many caveats you miss it.
• Gemini brings the largest context windows and the strongest Google ecosystem integration — Sheets, Drive, and BigQuery — though it occasionally references deprecated library functions.

The productivity payoff is real, too. McKinsey's State of AI survey found 80% of respondents report AI improved their individual productivity, and 50% say it helps them make better decisions. For analysts drowning in repetitive cleaning, charting, and syntax lookups, that's exactly the kind of work AI handles reliably.

The practical takeaway: point AI at tasks with checkable outputs — code generation, test selection, first-pass EDA — and you capture most of its value while sidestepping most of its risk. And if you want to push these narrow strengths further, agent skills are instruction packages that teach coding agents like Claude Code, Cursor, and Codex to handle specific analytical workflows. SkillzBench is building a free, independent directory of those skills — with security scanning and benchmarking of the lift each skill actually delivers — plus a free MCP connector to load them into any agent. Join the waitlist to be first in line when it launches. Skills, proven.

The Hallucination Problem: Plausible, Confident, and Wrong

The most dangerous AI errors don't look like errors. They look like a polished analyst's work — complete with numbers, citations, and confident prose. That's because generative AI doesn't actually "know" what is true or false; it only "knows" what fits the pattern, according to IBM. It optimizes for plausibility, not correctness, and hallucinations "can be reduced but not fully eliminated."

IBM identifies three distinct hallucination types that matter for anyone using AI on data:

• Factual hallucinations — inventing statistics, people, or events out of nothing.
• Contextual hallucinations — ignoring the context you actually provided, like Google's AI Overview once recommending nontoxic glue in pizza sauce based on a 2013 Reddit joke.
• Consistency hallucinations — contradicting itself within the same conversation, quietly undermining earlier conclusions.

The legal world has already learned this the hard way. In a widely cited 2023 case, ChatGPT produced fictitious case law — complete with plausible names, docket numbers, and quotations — for a US lawyer's court filings, per IBM's analysis. Every detail looked legitimate. None of it existed.

Data analysis carries the same risk in a different costume. Practitioner testing found that all three major AI assistants "can hallucinate statistics — meaning they sometimes generate numbers, citations, or study results that don't actually exist," according to KDnuggets. The failure is most likely exactly where business users rely on AI most: factual claims rather than code generation.

Complex reasoning compounds the problem. Stanford HAI's 2025 AI Index Report found that models "often fail to reliably solve logic tasks even when provably correct solutions exist, limiting their effectiveness in high-stakes settings where precision is critical."

And when AI agents work together, one hallucination can cascade. IBM describes a scenario where an agent fabricated a "no read activity in 90 days" metric for a storage bucket. Downstream agents, treating that fabricated number as verified fact, deleted the organization's only copy of financial records under legal hold. One invented metric; an irreversible, destructive outcome.

This matters now because agent adoption is accelerating — 40% of large organizations report scaling AI agents, up from 27% — while only about 20% of companies report mature frameworks for managing them. Verification checkpoints between agents aren't optional; they're the difference between automation and automated error. When building those pipelines, the skills you attach to each agent matter too — unvetted ones can carry their own risks, which is exactly why SkillzBench is building a free directory that scans, validates, and benchmarks agent skills before you connect them.

A Verification-First Workflow for AI-Assisted Analysis

If hallucination can't be eliminated, the answer isn't better prompts — it's better workflow design. McKinsey's research on AI high performers makes this distinction explicit: they "fundamentally redesign workflows that are enabled by AI rather than insert AI into existing ones," with roughly 75% redesigning workflows versus about 25% of everyone else.

Applied to data analysis, redesign means deciding up front where AI drafts and where humans verify — not asking a chatbot to reason from memory and hoping the numbers are real.

Ground every output in real data. IBM recommends retrieval-augmented generation to force answers from curated sources rather than the model's training patterns, since generative AI "only 'knows' what fits the pattern." In analytical work, the equivalent is a code-execution sandbox: upload the actual dataset and have the model compute results, not recall them. All three major assistants can run uploaded CSV and Excel files in sandboxed environments, which shifts the model's role from guessing statistics to executing code you can inspect.

Add explicit prompt guardrails. A simple instruction — "if unsure, say you do not know" — is one of IBM's core mitigation techniques. Practitioner testing found all three major assistants can hallucinate statistics, generating numbers and citations that don't exist, so confidence in the output is not evidence of correctness.

A verification-first workflow has four checkpoints:

• Ground the model in your real dataset via sandbox execution, never memory-based reasoning.
• Review all generated code before running it on important data — subtle statistical errors are a documented failure mode.
• Insert verification checkpoints between agents so one fabricated metric can't cascade into downstream decisions.
• Keep a human statistician as the final gate for anything informing a real decision.

That third checkpoint matters most as agentic adoption grows: 40% of large organizations are scaling AI agents, yet only about 20% report mature agent-management frameworks. IBM's cascading-hallucination example — where one fabricated metric led agents to delete legally protected financial records — shows what happens without checkpoints.

Finally, keep human statistical judgment in the loop. As the KDnuggets testing puts it, "no AI assistant replaces a solid understanding of statistics" — AI can pick a valid test while missing assumption violations like unequal variances or model mis-specification. If you're extending this workflow with agent skills, SkillzBench (skillzbench.com) is building a free directory and MCP connector so you can browse, verify, and benchmark skills before trusting them in your pipeline.

Putting It Into Practice: Guardrails, Agents, and Skills

The gap between AI's promise and AI's reliability doesn't close on its own — it closes through deliberate workflow design. The teams getting real results aren't the ones using the most tools; they're the ones deciding exactly where AI drafts and where humans verify.

That distinction matters because McKinsey's research shows high performers separate themselves through structure, not tool adoption. Roughly 75% of AI high performers redesigned their workflows around AI, compared with just 25% of everyone else — and while 80% of respondents report individual productivity gains, only 37% report EBIT contribution. Individual wins without redesigned workflows rarely reach the bottom line.

Start by drawing a hard line between drafting and deciding:

• Let AI generate Python/R code and run exploratory data analysis — tasks where practitioner testing found it genuinely strong.
• Require human verification of any output that informs a real decision, since all major assistants can hallucinate statistics — inventing numbers, citations, or study results that don't exist.
• Ground the model in your actual data via code-execution sandboxes rather than letting it reason from memory, following IBM's guidance on retrieval and guardrails like "if unsure, say you do not know."
• Keep a statistician in the loop for assumptions — independence, unequal variances, model specification — that AI won't check for you.

Agentic analytics pipelines deserve an extra layer of caution. Adoption is racing ahead: 40% of organizations with $1B+ revenue now report scaling AI agents, up from 27%, and roughly three in four companies plan agentic deployments within two years. Yet only about 20% of companies report mature frameworks for managing AI agents. IBM's documented example shows why that gap is dangerous — one agent fabricated a "no read activity in 90 days" metric, and downstream agents, treating the fabrication as fact, deleted the organization's only copy of financial records under legal hold. That's cascading hallucination: later agents don't just repeat the error, they build reasoning on top of it.

The practical defense is verification checkpoints between agents — and that's exactly the kind of workflow agent skills can automate. Because skills are instruction packages anyone can publish, quality varies enormously, which is why SkillzBench is building a free directory that security-scans, version-pins, and benchmarks each skill in a sandbox before you ever connect it. Its free MCP connector lets you attach only the skills you've chosen to each agent, so verification becomes part of the pipeline rather than an afterthought.

The takeaway is simple: AI drafts, humans decide. Where the two blur — especially in multi-agent pipelines — build explicit checkpoints, use vetted tooling, and never let plausibility masquerade as proof.