Can LLMs do predictive analytics?

The Prediction Problem: Why Businesses Are Tempted (and Burned)

Enterprise spending on LLM APIs more than doubled from $3.5B to $8.4B in just six months, and a big reason is that companies want these models to do more than write emails. According to market analysis from Future Market Insights, "domain-specific analytical workflows" are now a core growth driver as enterprises move from pilots into production.

That's the temptation: if an LLM can reason, summarize, and plan, surely it can forecast sales, demand, or churn. And sometimes it genuinely can — research shows LLMs predict time series well when data has clear trends and seasonality, and a NeurIPS 2024 paper found that integrating filtered news events with time series data produced "significant improvements in forecasting accuracy."

But the trap is real. The same market research warns that organizations deploying general-purpose LLMs without enterprise data grounding face significant accuracy and hallucination challenges. And a survey of LLM forecasting agents cites an uncomfortable finding from Tan et al. (2024): in ablation studies, removing the LLM component sometimes improved forecasting accuracy.

Why the gap between promise and performance? Three problems keep surfacing:

• Data type matters — LLMs excel on patterned, seasonal data but struggle with datasets lacking periodicity, and performance degrades as the number of periods grows source.
• Contamination masquerades as skill — benchmark gains may reflect memorized outcomes rather than real forecasting. As one researcher put it: "If a model is asked to forecast the 2022 midterms, and its training data goes up to 2023, it is not forecasting—it is recalling."
• Measurement itself is broken — the survey concludes that "measurement is a central limitation," with feature leakage and parametric leakage undermining most evaluations.

The stakes are escalating. Total enterprise generative AI spend hit $37B in 2025, up 3.2x from $11.5B in 2024, per Menlo Ventures' enterprise survey. Much of that flows into analytical use cases where a confident-but-wrong forecast carries real business cost.

This is why the "with vs. without" test matters so much. Any predictive analytics capability — whether a prompt technique, a tool-augmented agent, or a packaged agent skill — should be benchmarked against the same model running without it. That's exactly the discipline SkillzBench is building into its platform: testing every skill in a sandbox with the same model, with and without, and publishing the Lift in percentage points.

So the question isn't whether LLMs can do predictive analytics. The honest framing is: when do they actually help, and how do you know the gains are real? The rest of this article answers both.

What the Research Actually Says: Where LLMs Forecast Well (and Where They Fail)

The evidence on whether large language models can forecast is refreshingly specific — and it does not support a simple yes or no. Empirical testing of GPT-4 and GPT-3.5-turbo-instruct across ten real datasets found that LLMs excel on time series with clear patterns and trends but struggle when datasets lack periodicity. Accuracy also degrades as the number of periods grows, and the models are unusually sensitive to the most recent segment of the input sequence — add noise there and R² drops noticeably.

That pattern-based strength has a counterpart: on clean numerical extrapolation, the LLM itself can be the weakest link. A survey of LLM forecasting agents cites an ablation study in which removing the LLM component improved accuracy, and warns that benchmark gains may reflect contamination rather than genuine temporal reasoning. If a model's training data extends past the event it is asked to "forecast," it is not forecasting — it is recalling.

Where LLMs genuinely pull ahead is judgmental forecasting: reasoning over unstructured context before effects show up in the numbers. A NeurIPS 2024 paper found that integrating filtered news events with time series data produced significant accuracy improvements, which the authors describe as a potential paradigm shift. The survey agrees that this is the LLM agent's distinct edge, while specialized numerical models remain highly effective for pattern recognition in clean data.

Practically, that means the techniques worth testing look like this:

• Inject external human knowledge into prompts rather than relying on raw sequences alone
• Convert numerical sequences into natural language paraphrases before feeding them to the model
• Use agent-based filtering of news and policy documents alongside time series data
• Benchmark every predictive skill with the same model, with and without it, and publish the Lift separately

The stakes are rising quickly. Enterprise LLM API spend more than doubled from $3.5B to $8.4B in six months, and "domain-specific analytical workflows" now appear as a core growth driver in the enterprise LLM market. Yet organizations deploying general-purpose models without proper grounding still hit accuracy and hallucination problems — which is why testing before trusting matters more than model choice. SkillzBench is building exactly that: a free directory and MCP connector where skills are version-pinned, security-scanned, and benchmarked with and without the skill, so real Lift is visible before you deploy. Join the waitlist to test predictive analytics skills when the platform launches.

The Hidden Risk: Contamination, Leakage, and Overconfident Models

A model that "predicts" the 2022 midterms using training data from 2023 isn't forecasting — it's doing something far less impressive. That uncomfortable truth sits at the center of the LLM predictive analytics debate, and a recent survey of LLM forecasting agents states it bluntly: "If a model is asked to forecast the 2022 midterms, and its training data goes up to 2023, it is not forecasting—it is recalling."

The survey's conclusion is sobering: "measurement is a central limitation." Benchmark gains that look like genuine predictive skill may simply reflect data contamination — the model already knows the answer because it read about it during training. This means impressive-looking accuracy numbers on historical data can be an illusion.

Two specific leakage mechanisms make this worse:

• Feature leakage — lookahead bias, where post-event summaries or context sneak information about the outcome into the input.
• Parametric leakage — outcomes the model memorized during training, which it "recalls" rather than predicts.
• Reflexivity — deployed forecasts can change the outcomes being forecast, such as a predicted crash actually causing one.

There's a second problem: LLMs tend to be overconfident. A model that forecasts well but attaches misplaced certainty to its numbers is arguably more dangerous than one that fails visibly. The survey identifies calibration — techniques like verbalized confidence elicitation and post-hoc temperature scaling — as essential, alongside contamination-resistant live evaluation before you trust any predictive output.

The stakes are rising fast. Enterprise LLM API spend more than doubled from $3.5B to $8.4B in six months, and organizations are pushing these models into production analytical workflows where hallucination and accuracy challenges are well documented.

The practical implication for anyone deploying a predictive analytics skill or agent is simple: demand a with-and-without test. Run the same model on the same tasks, once with the skill and once without, and measure the Lift in percentage points. The survey cites ablation research where removing the LLM component actually improved accuracy — proof that "it has an LLM in it" is not evidence it helps.

This is exactly the discipline SkillzBench is being built to enforce: every skill version-pinned to a commit SHA, tested in a sandbox with the same model, with and without the skill, with Lift published separately from ratings. Before you trust any agent's numbers, verify the gain is real — not memorized.

How to Make LLM Predictions Work: Practical Techniques and Skill Testing

The difference between an LLM forecast that works and one that embarrasses you often comes down to three practical techniques, not model choice. Researchers testing GPT-4 and GPT-3.5-turbo-instruct across ten real datasets found that "incorporating external knowledge and adopting natural language paraphrases positively affects the predictive performance of LLMs for time series." In plain terms: how you present the data matters as much as the data itself.

Three techniques consistently move the needle:

• Inject external knowledge into prompts. Give the model context it cannot infer from raw numbers — product launches, policy changes, seasonal events — so it reasons rather than guesses.
• Paraphrase numbers into natural language. LLMs treat numerical values as text, so converting sequences into descriptive prose helps them latch onto trends and seasonality.
• Pair agent-based filtering with time series. A NeurIPS 2024 paper showed "significant improvements in forecasting accuracy" by integrating filtered news events with numerical data, calling it "a potential paradigm shift" in the field.

That third technique is where agent skills come in. A skill is simply a packaged set of instructions — exactly the kind of prompt engineering, filtering logic, and paraphrasing rules described above — that any agent can load and reuse. Instead of rebuilding the workflow each time, you install a skill and the agent handles the unstructured-data side of forecasting for you.

But packaged instructions are also a supply chain risk. Published research found that roughly 26% of agent skills carry at least one vulnerability, which means "does it work?" is only half the question — "is it safe?" is the other half. This is the problem SkillzBench is being built to solve. Before any skill ships, the plan is to benchmark it in a sandbox using the same model with and without the skill, then publish the Lift in percentage points so you can see whether the skill actually helps. Every version gets pinned to a commit SHA, run through security scanning, and re-benchmarked when upstream changes land.

Why does the "with and without" test matter so much for forecasting specifically? Because a survey of LLM forecasting agents found that measurement is a central limitation — benchmark gains sometimes reflect memorized outcomes rather than genuine prediction, and one cited ablation study showed that removing the LLM component can actually improve accuracy. A skill that looks impressive in a demo may be adding nothing, or worse, actively hurting your numbers.

The takeaway for anyone deploying predictive analytics with agents: demand evidence. Look for skills that have been tested with a clear before-and-after comparison, security-scanned, and version-pinned — and when SkillzBench launches, you'll be able to browse and test skills free, with Lift scores shown separately from every rating.

Your Next Step: Test Before You Trust

So, can LLMs do predictive analytics? Yes — but only where their strengths actually lie, and only after you've proven it. The evidence points to a clear division of labor: LLMs shine at judgmental forecasting over unstructured context, while specialized models still win on clean numerical extrapolation.

A survey of LLM forecasting agents puts it plainly: specialized numerical models remain highly effective for pattern recognition in clean time-series data, but LLM agents offer a distinct advantage in judgmental forecasting — reasoning over news and policy documents before those effects appear in the numbers. And NeurIPS 2024 research showed significant accuracy gains when filtered news events are integrated with time series data.

Here is the decision framework to take away:

• Use LLMs for judgmental forecasting — blending unstructured context like news, filings, and policy signals with your data.
• Keep specialized statistical and foundation models for clean numerical extrapolation on well-behaved time series.
• Demand with/without testing for any predictive skill you adopt — the same survey cites research where removing the LLM component actually improved accuracy.

That last point matters most. The same survey warns that "measurement is a central limitation," because benchmark gains may reflect data contamination rather than genuine forecasting. As it memorably puts it: if a model is asked to forecast the 2022 midterms and its training data goes up to 2023, it is not forecasting — it is recalling. Any predictive skill you adopt must be tested with and without the skill, on the same model, before you trust it.

The stakes are rising fast. Enterprise LLM API spend more than doubled from $3.5B to $8.4B in six months, and total enterprise generative AI spend hit $37B in 2025 — up 3.2x year over year. With domain-specific analytical workflows now a core growth driver, untested predictive skills are a real business risk, not a theoretical one.

This is exactly the gap SkillzBench is being built to close. The platform will benchmark every skill in a sandbox with the same model, with and without the skill, and publish the Lift in percentage points — the contamination-resistant, version-pinned testing the research says forecasting evaluation needs. And since published research has found that roughly 26% of skills carry at least one vulnerability, security scanning and license checks come standard alongside the benchmarks.

Test before you trust — and let the Lift decide. Join the free SkillzBench waitlist to benchmark predictive analytics skills at launch, and browse the free skills directory and free MCP connector to equip your agents with forecasting skills that have earned their numbers.