How to Extract Data from PDFs with AI: A Practical Guide

Why PDFs Break AI Agents (And Why the Model Isn't to Blame)

Your AI agent confidently reads a table from a PDF — and half the numbers are wrong. Before you blame the model, look at what the model actually received: not a table at all, but a flattened stream of text where rows, columns, and headers have dissolved into noise.

The root cause goes back decades. PDFs were designed in the early 1990s to preserve how a page looks, not what it means — they store positioned text and drawing instructions, with no concept of rows or columns. As one technical analysis puts it, that single detail explains most PDF table parsing failures. A PDF is a digital snapshot of a page, not a true digital document with semantic meaning.

This matters because most AI agent pipelines hand the model whatever the extraction layer produces. When that layer flattens structure, three predictable failures follow, per Nutrient's 200-document benchmark:

• Tables become "word soup" — cell values lose their row and column relationships entirely
• Headings vanish into body text, so the model can't tell where one section ends and another begins
• Reading order gets scrambled, mixing multi-column layouts into incoherent sequences

The benchmark numbers are stark. PDF.js — the default extractor in many agent frameworks, built for browser rendering rather than structure recovery — scored exactly 0.000 on table structure recovery and 0.000 on heading detection across 200 real-world documents. Yet the same models, given properly structured input, perform dramatically better.

So the model isn't hallucinating in the usual sense. It's doing its best with garbage input — and the fix isn't a better model, it's a better extractor. Upgrading the LLM won't help: NVIDIA's research found that moving from an 11B to a 90B vision model didn't even improve retrieval recall, because the bottleneck was upstream of the model.

The consequences are well documented. Independent testing found error rates of 1–6% scattered randomly across every column, plus fabricated values and cross-record contamination. And schema enforcement won't save you — as one structured-output analysis notes, a schema guarantees the shape of your JSON, never the truth of its contents. One documented case saw a model return "quantity": 1 when the correct value was 0.46. The JSON validated; the invoice total did not.

The takeaway for anyone building agent workflows: diagnose your extraction layer before swapping prompts or models. If you're assembling a document-processing agent, validated extraction skills and tooling matter more than model choice — SkillzBench is building exactly this kind of independent verification and benchmarking for agent skills, so you can test what actually works before it touches your data.

The Repeatable Extraction Strategy: Classify, Route, Validate

Stop hunting for the one tool that handles every PDF — it doesn't exist. As StackAI puts it, reliable extraction "isn't about finding the perfect tool. It's about having a repeatable strategy," because tool performance varies dramatically by document type: a library that excels on research papers may fail on bank statements.

That strategy has three stages: classify, route, validate.

Classify first. Before any extraction, determine whether each PDF is digital-born (you can highlight and copy its text) or scanned (the page is an image requiring OCR). Many real-world PDFs are hybrids, so don't assume. This single check determines which downstream method will actually work.

Route by document type. The decision tree from StackAI is straightforward:

• Scanned documents: run OCR followed by table structure recognition.
• Digital tables with visible grid lines: use lattice parsing.
• Digital borderless tables with strong column alignment: use stream parsing.
• Complex or varied layouts at scale: use ML-based table detection.

Routing matters because a one-size-fits-all approach fails quietly. When your AI agent returns wrong table data, the cause is usually an extraction layer that flattened structure before the model saw it — "the fix isn't a better model. It's a better extractor," as one benchmark of 200 real-world documents concluded. The same benchmark found PDF.js, the default extractor in many agent frameworks, scored exactly 0.000 on both table structure recovery and heading detection.

Validate with a schema and a golden set. Ask the model for JSON that strictly follows a defined schema with explicit field names and types, then validate completeness and retry failed records — typically valid by the second or third attempt, per empirical testing by OpenNews, which found 1–6% error rates scattered randomly across every column. But here's the critical warning: schema enforcement guarantees shape, never truth. One documented case saw a model return a perfectly valid "quantity": 1 when the correct value was 0.46 — the JSON validated, the invoice total did not. Grounding (citations, bounding boxes) is what lets agents verify their own answers, and keeping bounding box metadata turns debugging from guesswork into lookup.

Finally, build a golden set of ~20 representative PDFs covering your real-world variety, with checks like column-count consistency, a ≥95% numeric parse rate, and totals verification. Re-run it before any pipeline change.

Once extraction is solid, agent skills can automate the downstream workflow — validation, approval routing, reconciliation. SkillzBench is building a free directory and free MCP connector to help you find, verify, and benchmark those skills before they touch your documents.

Choosing Your Model and Proving It Works: Golden Sets and Accuracy Reality

Which LLM should you trust with your invoices? The honest answer is that no single model wins everywhere — and the benchmarks that say otherwise are usually selling something. That's why choosing a model and proving it works are really the same task.

One useful data point comes from Koncile, which tested 300 text-based and 200 scanned invoices across the three major models. On digital PDFs, GPT hit 98% accuracy, Claude 97%, and Gemini 96%. On scanned invoices the order flipped: Gemini led at 94% thanks to its native vision, while GPT (91%) and Claude (90%) both required external OCR. The same comparison found each model has a distinct personality — GPT for versatility and price/performance, Claude for format rigor and compliance (it doesn't train on your data by default), Gemini for multimodal speed.

Treat those numbers with skepticism, though. Independent benchmarks like OmniDocBench and olmOCR-Bench repeatedly fail to match vendors' self-reported accuracy — even Mistral concedes a single aggregate number "can both understate and overstate real-world performance." Meanwhile, an independent OpenNews investigation found error rates of 1% to over 6% per column scattered randomly across rows, plus hallucinated data — a sobering reality check on any headline accuracy figure.

So run your own test. The core practice is a golden-set regression suite:

• Assemble roughly 20 representative PDFs covering your real-world variety — different layouts, scanned and digital, messy edge cases.
• Add validation checks: column count consistency, header similarity across pages, and totals/checksum verification.
• Require a numeric parse rate of at least 95% on numeric columns before trusting output.
• Re-run the suite before every pipeline change, so regressions surface immediately.

Two more habits make the suite trustworthy. First, remember that schema enforcement guarantees shape, never truth — a JSON payload can validate perfectly while containing a wrong quantity. Grounding via citations or bounding boxes lets agents verify their own answers, and per StackAI's advice, keeping bounding box metadata turns debugging from guesswork into lookup. Second, remember the OpenNews test predates modern models; your own numbers on your own documents are the only ones that count.

Once your golden set passes, agent skills can automate the rest of the workflow — validation, approval routing, reconciliation. SkillzBench is building a free, independently benchmarked directory of agent skills plus a free MCP connector, so you can see which skills actually lift accuracy before you wire them in.

From Extracted Data to Agent Workflows: Where the Real Value Lies

Getting the data out of a PDF is the easy part. The real return on your extraction investment comes from what happens after — when structured output flows into downstream agent workflows like validation, approval routing, and reconciliation without a human handoff.

As V7 Labs puts it, extraction is only one part of the puzzle. AI agents can pull key entities from an extracted invoice, trigger approval routing, run three-way PO matching, and flag discrepancies against internal systems automatically. That's where business value compounds — and where extraction costs become trivial.

And those costs are genuinely low. In testing across 500 invoices, extraction averaged €0.015–€0.025 per invoice, with GPT-4o processing 1,000 invoices for about $17.50. When a single invoice costs less than two cents to structure, the economics of automating the entire accounts payable chain become hard to ignore.

There's one pricing trap worth flagging. Hyperscalers charge roughly $1.50 per 1,000 pages to read a page — but $30 to $50 per 1,000 pages to structure it, a 20x–33x premium, according to Firecrawl's pricing comparison. Budget accordingly if you're evaluating cloud document APIs.

Once structured data lands in your systems, agent workflows typically include:

• Validation — checking extracted totals against computed line-item sums and flagging schema failures for retry
• Approval routing — sending invoices above threshold values to the right approver automatically
• PO matching — comparing invoice lines against purchase orders and receipts to catch discrepancies
• Reconciliation — feeding clean, structured records into dashboards and accounting systems

Building these workflows is where agent skills matter. SkillzBench is building a free directory of vetted agent skills — with security scans, sandbox benchmarks, and trust scores — plus a free MCP connector so you can attach the skills you choose to any agent. If you're assembling a document-processing pipeline, it's worth browsing what's available before writing every workflow from scratch.

The pattern to remember: extraction quality determines whether your downstream agents can be trusted, but the extraction itself is never the finish line. Treat it as step one of a pipeline, validate aggressively, and let agents handle the workflow that follows.

Go Further with Agent Skills: Build Your PDF Pipeline on Verified Building Blocks

By now, the pattern should be clear: the hard part of PDF extraction isn't the model — it's everything wrapped around it. Classification, routing, structure-aware extraction, validation loops, golden-set testing. Hand-coding every one of those steps for every document type is a serious engineering commitment, and practitioner guidance is blunt that a repeatable strategy, not a perfect tool, is what separates reliable pipelines from fragile ones.

That's where agent skills come in. Instead of writing every step from scratch, you can assemble your pipeline from pre-built instruction packages — SKILL.md files that teach coding agents like Claude Code, Codex, Cursor, and Gemini CLI how to handle document classification, table extraction, schema validation, and output checking. Think of them as verified building blocks: each one encodes a proven workflow step so your agent doesn't have to reinvent it.

The catch is trust. Skills are everywhere, but published research found that roughly 26% of agent skills carry at least one vulnerability, and a skill that quietly exfiltrates data or injects bad instructions can undo all the careful validation you built downstream. That's worse than a failed extraction — as one analysis of extraction tools notes, schema enforcement guarantees shape, never truth, and the same applies to any instruction package you bolt into your pipeline.

This is the problem SkillzBench is being built to solve. The platform — currently in development, with a waitlist live — is designed around five pillars for exactly this use case:

• Discover — continuously crawl and dedupe skills from GitHub so you're not guessing which extraction or validation skills exist.
• Verify — pin every version to a commit SHA, then license-check, security-scan, and quality-judge it before it reaches your pipeline.
• Benchmark — test every skill in a sandbox with the same model, with and without the skill, and publish the Lift in percentage points.
• Equip — connect via a free MCP server, toggling skills and tools per agent so each agent only sees what you chose for it.
• Stay current — track upstream changes, re-scan, re-benchmark, and notify you when a skill shifts.

The benchmarking piece matters most for PDF work. Independent benchmarks already disagree sharply with vendor self-reported accuracy — OmniDocBench testing showed a 0.9B-parameter model topping table extraction at 94.76 TEDS while GPT-5.2 scored 82.95, proving that neither size nor popularity predicts performance. Skill-level Lift benchmarks apply that same discipline to instruction packages, and SkillzBench keeps Lift scores strictly separate from ratings, with advertising revenue never influencing either. Everything — directory, MCP connector, benchmarks, alerts — is free.

Build your PDF pipeline on blocks someone else has already scanned, pinned, and measured.