AI SQL: Getting Language Models to Write Queries That Run

The Accuracy Cliff: When Benchmarks Lie About Your Database

The numbers behind AI-generated SQL look spectacular right up until they touch a real database. GPT-4o scores 86.6% on Spider 1.0, the academic text-to-SQL benchmark, but collapses to just 10.1% on Spider 2.0's real enterprise workloads — an 8.5× drop one analysis calls the "accuracy cliff". MIT's BEAVER benchmark, built from actual private enterprise warehouse logs, is even harsher: off-the-shelf LLMs, including GPT-4o, scored essentially zero.

The reason isn't model capability — it's missing context. The decisive information for any real query lives outside the database entirely: how "active customer" is defined, which join paths are trusted, when your fiscal year starts. Academic benchmarks use 10–20 tables with 50–100 columns; enterprise estates routinely exceed 1,000 columns across 100+ tables, and the model has never been exposed to your specific schema's semantics. As the paper's own author put it, database knowledge doesn't emerge with scale — models have to be deliberately given the information.

Real deployments confirm the gap. Snowflake's own engineering team measured GPT-4o at 51% accuracy on 150 internal BI questions — roughly every second answer wrong. Uber's internal text-to-SQL systems overlap with the correct tables only half the time, and just 10–20% of AI-generated answers on their heterogeneous systems are accurate enough to drive business decisions.

What makes this dangerous is that failing queries rarely announce themselves. A query that runs and returns data looks identical to a correct one, and the most damaging errors are silent failures — queries that execute successfully while feeding critical reports wrong numbers. Think double-counted joins, missing filters that include test data, or a "Labor Day" calculation that assumes the wrong country's calendar. The financial stakes are real: Gartner estimates poor data quality costs organizations an average of $12.9M per year, and a BlackLine survey found roughly 7 in 10 finance professionals made a significant decision based on incorrect or outdated data.

The failure modes are predictable enough to catalog:

• Metric drift — queries that quietly diverge from approved business definitions
• Wrong joins that inflate counts through duplicated rows
• Schema guessing — inventing tables or columns that don't exist
• Calendar mismatches between fiscal and standard dates
• Governance bypass, including PII leakage, with no audit trail

The takeaway for any team adopting AI SQL: executable does not mean correct. Even the best model in a 19-model benchmark scored only ~56/100 on exactness against a human baseline, which is why SQL skills remain necessary. Teams wiring agents into their data workflows should treat verification tooling as part of the stack — the free agent skills directory and MCP connector being built at SkillzBench are designed to make it easier to find and test skills for exactly these kinds of validation workflows.

The Silent Failure Problem: When 'It Ran' Isn't 'It's Right'

A query that runs and a query that's right look identical on screen — and that's exactly what makes AI-generated SQL dangerous. The database engine never raises an error for a join that double-counts revenue or a filter that quietly includes test records.

Tinybird's benchmark of 19 models against a 200-million-row GitHub events dataset makes this concrete. The best performer, Claude 3.7 Sonnet, produced 100% valid queries yet scored only ~56/100 on exactness — meaning even the top model got nearly half the answers substantively wrong. Most models also read 1.5x–2x more rows than a human would, and one, o4-mini-high, read 32% more data than the human baseline despite generating valid SQL for all 50 prompts (Tinybird's benchmark).

The failure patterns are predictable enough to catalog:

• Double-counting joins — models miss contextual details like issue numbers not being globally unique, inflating counts
• Missing filters — test data, date ranges, or exclusions the prompt never mentioned get skipped entirely
• Fabricated metrics — plausible-sounding business calculations that match no approved definition
• Over-reading — when uncertain, models "overdo it," selecting more fields and broadening filters

As one enterprise analysis puts it, silent failures pose greater danger than errors precisely because they appear correct while delivering incorrect business insights. A query that parses and executes looks the same as a correct one; no runtime check catches a logically flawed join. The stakes are real — a BlackLine survey found roughly 7 in 10 finance professionals believed they or their CEO made a significant decision based on incorrect or outdated data.

The mindset shift this demands is simple but non-negotiable: treat successful execution as the start of verification, not the end of the workflow. Compare result sets against known-good answers, lint generated SQL before it runs, and feed execution results back to the model. Teams building this verification loop into agentic workflows can browse and test free agent skills — including SQL validation and data-analysis skills — on SkillzBench's pre-launch directory, where each skill's measured Lift is shown alongside its trust score. Whatever tooling you choose, the principle stands: "it ran" is the beginning of the question, never the answer.

Engineering Context: The Prompting Patterns That Actually Work

If the accuracy cliff is a context problem, then the fix is context engineering — not a bigger model. The research is unambiguous on this point: as AWS Applied Scientist Shuaichen Chang put it, database knowledge is "not something that as the model becomes more powerful, they can naturally get" — it has to be supplied explicitly (Arize interview).

Four levers carry most of the weight. First, inject the schema with foreign-key relationships spelled out. Chang's experiments showed that including table relationships "consistently" improves zero-shot performance, and adding sample database content — actual values from the tables — was "even more beneficial." One caveat: models are picky about how that content is represented, responding differently to row-wise versus column-wise layouts.

Second, add sample values. This matters because many failures are semantic, not syntactic: Tinybird's 19-model benchmark found models missing contextual details like issue numbers that aren't globally unique, causing wrong joins and inflated counts (Tinybird benchmark).

Third, use 2–3 annotated few-shot examples. This is the most cost-effective finding in the entire research base: in-context learning with a handful of demonstrations (fewer than 32) can be "comparable or even better than fine-tuning-based methods" trained on 7,000 examples on Spider. Even cross-domain examples — pairs from a different database — beat zero-shot when no in-domain pairs exist yet.

Fourth, lint and validate before anything executes. Tinybird's production recommendations include query linting as post-processing, feedback loops that feed execution results and error messages back to the model, and structured output with self-evaluation checks.

The escalation path matters too. AWS recommends starting with prompt engineering before fine-tuning "because this allows rapid iteration without data collection," escalating only if needed (AWS best practices). Their effective prompt pattern is simple: a clear instruction that SQL is wanted, relevant schema details, 2–3 annotated example pairs, and retrieval of similar examples.

For teams mapping their maturity, Promethium's "Five Levels of Context" framework connects context depth directly to accuracy:

• Technical metadata only — roughly 10–20% accuracy
• Relationships and keys — 20–40%
• Catalog and business definitions — 40–70%
• Semantic layer — 70–90%
• Tribal knowledge and feedback loops — 90%+

Treat those vendor-positioned numbers as directional rather than guaranteed, but the pattern holds across sources: the more context you supply, the closer production gets to benchmark performance. Teams running agentic SQL workflows can package these prompting patterns as reusable instruction sets — SkillzBench is building a free directory and MCP connector for exactly that kind of verified, benchmarked agent skill.

Building the Feedback Loop: From One-Off Prompts to a Working System

A one-off prompt that happens to work is not a system — it's luck. The teams getting real value from AI-generated SQL treat every query as part of a loop: generate, execute, compare against known results, and feed what failed back into the next attempt.

The first discipline is judging correctness the right way. As AWS's Shuaichen Chang explains, the same intent can be written many ways, so you should judge by execution accuracy — comparing result sets against ground truth, not eyeballing the SQL (per the Arize paper-reading discussion). This matters because silent failures are the defining risk: queries that execute cleanly but return wrong numbers look identical to correct ones, as Promethium's enterprise analysis makes clear. When a query fails, pass the actual error message back to the model — Tinybird's production guidance recommends exactly this feedback loop with execution results.

Second, don't assume your verification is stable. Temperature 0 is not deterministic — Anthropic's own documentation states results "will not be fully deterministic" even at 0.0, and research cited by Colrows found accuracy variations up to 15% across naturally occurring runs. Repeat your verification, and don't trust a single passing run.

Third, give the loop something to work from. AWS's best practices for Text2SQL recommend:

• A central data catalog with documented business definitions, so the model picks the right tables and columns
• Golden queries — 2–3 annotated natural-language-to-SQL example pairs per prompt
• Materialized views for common queries, which simplify generation and avoid complex SQL and performance issues
• Cached parsed SQL and monitoring of each generation stage

The business context is the part the model can never infer. Metric definitions, join paths, and fiscal-calendar rules live in documentation and analysts' heads — which is exactly why GPT-4o drops from 86.6% on Spider 1.0 to roughly 10% on Spider 2.0's real enterprise workloads. Your catalog closes that gap.

Finally, keep a human in the loop. Tinybird's 19-model benchmark found even the best model scored only ~56/100 on exactness, and its conclusion is blunt: SQL remains a necessary skill — the model writes, the human verifies. Teams extending this workflow with agents can browse and test free agent skills and the MCP connector SkillzBench is building (join the waitlist at skillzbench.com), but the verification discipline stays yours.

Scaling AI SQL Safely: Agent Skills, Efficiency, and What Comes Next

Getting AI-written SQL into production is not a research problem anymore — it's an engineering and economics problem, and both halves now have workable answers. The context-provision patterns that close the accuracy gap (schema injection, few-shot examples, validation-before-execution) are proven, and the cost of running them at scale is falling fast.

Google Cloud's proxy-model approach illustrates the economics side. LLM calls embedded inside SQL normally add 10–100x latency and roughly 1000x cost, but training a lightweight classifier on about 1,000 LLM-labeled sample rows cut a typical 1M-row query to ~400x fewer tokens with 30x–100x latency reduction — while preserving 90%–116% of the LLM's F1 across benchmarks. Production-grade AI SQL is affordable if you architect for it.

The correctness side is where teams should focus their packaging effort. The patterns that work are consistent: inject foreign keys and sample values, provide 2–3 annotated example pairs, lint queries before execution, and feed errors back. AWS's guidance to start with prompt engineering before fine-tuning holds because it enables rapid iteration without data collection. These are exactly the kinds of repeatable instructions that fit naturally into agent skills — SKILL.md packages that give coding agents like Claude Code, Cursor, or Gemini CLI the schema documentation, prompting patterns, and validation rules your team has already proven.

• Schema and foreign-key context in every prompt, since models can't infer your database's structure
• 2–3 in-domain NL-to-SQL example pairs, which research shows can rival fine-tuning on thousands of examples
• Query linting and validation before anything executes, catching hallucinated tables and missing filters
• Feedback loops that return execution results and error messages to the model

One caution for teams going the agent-skills route: install with your eyes open. Published research found that roughly 26% of published agent skills carry at least one vulnerability, and a skill that quietly reshapes how your agent writes SQL against production databases is a meaningful attack surface. Verify what you install — check provenance, licenses, and whether the skill has actually been tested, not just downloaded.

That verification burden is exactly what SkillzBench is being built to remove: a free, independent directory where every skill is pinned to a commit SHA, security-scanned, and benchmarked in a sandbox — with and without the skill — so you can see the Lift before you deploy it, plus a free MCP connector for per-agent loadouts. It's pre-launch now, with the waitlist live.

The trajectory is clear. Silent failures — queries that run but return wrong numbers — remain the defining risk, but proven prompting patterns, cheap proxy models, and verifiable, reusable skills are converging on a stack where AI SQL you can actually trust becomes the default, not the exception.