Can ChatGPT solve Erdős problems?

The Headline vs. The Baseline: What AI Actually Solves

In recent months, AI has seemingly cracked problems that stumped mathematicians for decades — yet the same systems fail most research-level mathematics they attempt. That gap between headline and reality is where honest expectations live.

The headlines are real. OpenAI announced a disproof of the 80-year-old Erdős unit distance conjecture, a result Fields Medalist Tim Gowers called "a milestone in AI mathematics." Google reported its AI solved nine open Erdős problems, two of which had been open for over 50 years. And in a 24-hour controlled challenge, the ProofCouncil team solved 6 of 10 open problems using public models, per their evaluation paper.

The baselines are sobering. State-of-the-art LLMs achieve only about 10–15% accuracy on research-level lemma proving, according to the LemmaBench benchmark. Even OpenAI's flagship unit-distance result succeeded only ~50% of the time — at maximum token budget, on an internal model more capable than anything available to the public.

The Navier–Stokes effort shows what "solving" actually costs. OpenAI's run used roughly 10,000 concurrent agents over 88 hours, generating 2.7 million agent messages and an estimated ~$10 million in compute — then required another 17 hours to formalize the proof in Lean. No public ChatGPT model can replicate it.

Three lessons transfer directly to anyone deploying agents on hard problems:

• Scaffolding beats prompting. Every success used multi-agent architectures: author-critic loops, parallel problem variants, and cross-pollination of insights between agent groups — never a single prompt.
• Verification is non-negotiable. Credible results were checked via Lean formalization or independent expert review, and the top failure mode was quietly solving an easier version of the problem, not producing wrong math.
• Transparency enables calibration. Automated-reasoning researcher Marijn Heule notes that "the failures are exactly the information needed to understand the method's limitations" — results you can't independently verify tell you little.

So: can ChatGPT solve Erdős problems? Sometimes, with enormous resources, heavy scaffolding, and expert verification — usually no, and never with one prompt. The same calibrated expectations apply to agents on any demanding workflow. That's why building verification into your agent stack matters from day one — it's the principle behind what SkillzBench is building with its free skills directory: benchmarking every skill in a sandbox, with and without it, so you see the measured lift before you trust it.

Why Agentic Architecture Wins Where Single Prompts Fail

Look at the results that actually landed, and a pattern emerges: none of them came from a single clever prompt. The ProofCouncil evaluation solved 6 of 10 open problems using an author-critic loop where one agent writes and a council of independent models critiques it, with the critic reset every three rounds for fresher review.

OpenAI's Navier–Stokes effort scaled the same ideas dramatically. Roughly 10,000 concurrent agents ran in communicating groups, each prompted with a different variant of the problem — proof and disproof directions pursued simultaneously — with insights cross-pollinated between groups. The team also staged escalation, first solving the easier Euler regularity problem with about 100 agents over 50 hours, then feeding that resolution into the harder Navier–Stokes agents. Formal verification in Lean closed the loop, with an additional 17 hours of formalization checking the result.

The architecture matters more than raw model power. Notice what these systems did well: they ground through a tedious proof strategy a human had abandoned because it "consumes much time and frequently doesn't work out," and they combined techniques from algebraic number theory that no single mathematician could hold in one head. That is brute-force persistence plus cross-domain knowledge — not human-style creativity. Tim Gowers called the unit-distance result "a milestone in AI mathematics," but it succeeded only about 50% of the time even at maximum token budget, and general-purpose LLMs still manage only ~10–15% accuracy on research-level lemma proving.

Just as instructive is where agents fail. Across the credible results, the dominant failure mode was not producing wrong math — it was misinterpreting the problem, solving an easier or subtly different question than the one asked. That is why verification is non-negotiable: Lean formalization, independent expert review, or proof certificates caught the gap between what the agent produced and what was actually required.

The lesson transfers directly to agent design on any hard workflow:

• Run author-critic loops rather than trusting a single generation, and reset the critic periodically.
• Pursue competing approaches in parallel instead of committing to one path early.
• Solve an easier adjacent problem first and feed the insight into the harder one.
• Verify outputs through an independent channel before trusting them.

This scaffolding is exactly what agent skills make practical for non-researchers — instruction packages that give an agent the right tools and guardrails for a specific job. Since published research found roughly 26% of community skills carry at least one vulnerability, knowing which building blocks are safe matters as much as assembling them, which is the gap SkillzBench's free directory, sandboxed benchmarking, and MCP connector are being built to close. Agents win on hard problems by playing to machine strengths under heavy scaffolding — and the same calibrated expectation applies whether the task is a conjecture or your quarterly reporting pipeline.

Verification and Transparency: The Non-Negotiable Steps

The most trustworthy results in this story are trustworthy precisely because nobody took them on faith. OpenAI's Navier–Stokes resolution was formalized and verified in Lean over an additional 17 hours after the agent swarm finished, according to OpenAI's own account. In the ProofCouncil evaluation, one of the five correct solutions was externally verified by three independent experts.

This is the pattern worth internalizing: every credible result was independently checked. Marijn Heule (CMU) notes that computer proofs went "from the least trustworthy thing in mathematics to the most trustworthy, but at an enormous cost" — and that cost is justified. Correctness is necessary, and so is attribution.

Heule's critique of OpenAI's announcements is blunt. The claimed capability cannot be verified because the model, prompts, problem selection, failures, and compute costs were all undisclosed, as he argues in a CACM analysis. His standard for any capability claim is simple:

• Disclose the model and the prompts used
• Report the failures, not just the wins — "the failures are exactly the information needed to understand the method's limitations"
• Publish verification artifacts so others can check the work
• Resolve attribution questions before announcing, not after

The attribution question is not hypothetical. The BBC's reporting documents NYU's Tristan Buckmaster alleging OpenAI began its Navier–Stokes work only "after information about our work had reached OpenAI," with OpenAI admitting it "cannot rule out" that de-identified user data helped improve its models. The contrast with DeepMind's AlphaTensor — whose result was surpassed within a week by conventional search once verification code was released — shows why openness matters. Openness is what lets outsiders calibrate whether a result reflects new capability or good marketing.

The business translation is direct. An agent's output — a proof, a report, a code change — earns trust only after independent verification, never before. Verification is not overhead; it is the product. And when a vendor claims an agent can do something, ask for the failure data first, because that is exactly what tells you the method's limitations.

This is the same philosophy behind SkillzBench's approach: benchmark every skill in a sandbox with and without it, publish the Lift separately from any rating, and never let money touch scores. Failure data is a feature, not an embarrassment. Any agent workflow you deploy — combinatorial or otherwise — deserves the same treatment.

The Real Near-Term Win: Accessibility, Not Replacement

When a Fields Medalist like Tim Gowers first heard about AI cracking an 80-year-old Erdős conjecture, he admits he "spent the evening adjusting my world view" and wondered "maybe it would be all over for mathematicians very soon," as Ars Technica reports. His relief came when he learned it was a disproof — and his medium-term outlook has settled on something more interesting than replacement: complementarity, where AI handles curated problem lists while humans choose which questions matter.

The clearest articulation of where the real value lies comes from Marijn Heule (CMU), whose CACM analysis identifies the "encoding bottleneck" as the thing LLMs are actually breaking open. This is the expert-only craft of reformulating a conjecture so a solver can attack it — a skill that historically required years of automated-reasoning training.

The proof case is Giles Gardam. In 2021, a group theorist with no automated-reasoning pedigree refuted Kaplansky's unit conjecture simply by encoding it for a SAT solver. Heule's vision of "a thousand Gardams" — LLMs translating hard problems into solver-ready forms for every domain expert — would be transformative, in his words, "even if the underlying capabilities are not new."

This maps directly onto how businesses should think about agents on hard problems:

• The win is accessibility: expert-level machinery placed within reach of non-specialists who know their domain best.
• Expectation-setting matters: even OpenAI's internal model solved the unit-distance problem only ~50% of the time at maximum token budget, and SOTA public LLMs manage just ~10–15% on research-level lemmas.
• Verification stays non-negotiable — every credible result in this space was checked by Lean, expert review, or proof certificates.

The catch is that "within reach" cuts both ways. Agent skills — the instruction packages that give coding agents and chat assistants their specialized capabilities — are exactly this kind of expert machinery packaged for non-experts. But published research found ~26% of skills carry at least one vulnerability, and nobody hands you a scorecard.

That's the gap SkillzBench is being built to close: a free directory where every skill is version-pinned, security-scanned, and benchmarked in a sandbox with and without the skill, so you can see the measured Lift before you trust it — plus a free MCP connector to equip agents with only what you've chosen. The near-term revolution isn't AI replacing your experts. It's your experts finally aiming the machinery — with the right skills attached.

How to Run Agents on Hard Problems: A Practical Playbook

The gap between "AI solved a famous open problem" and "AI solved your problem" is mostly architecture. Every credible result in this space came from scaffolding — not one-shot prompts — and that scaffolding is something you can copy.

Start with a loop, not a prompt. The ProofCouncil system that solved 6 of 10 open problems in a controlled challenge used an author-critic loop, an LLM council of different models, and a compute node running SageMath and GAP — with the critic reset every three rounds to keep its reviews independent (per the arXiv evaluation). A single model, asked once, reliably produces the ~10–15% pass rate that research-level lemmas actually measure (LemmaBench found).

Next, run parallel variants and escalate. OpenAI's Navier–Stokes effort organized roughly 10,000 concurrent agents into groups, each prompted with different problem variants covering both proof and disproof directions, and cross-pollinated insights between groups. Crucially, it staged escalation: nearly 100 agents first spent about 50 hours on the easier Euler regularity problem, and that resolution fed into the harder attack (OpenAI's writeup describes the full pipeline).

A distilled playbook:

• Scaffold with multi-agent loops — author, critic, and specialist tools — instead of one-shot prompts.
• Run parallel problem variants, including "try to disprove it" branches, and share insights across them.
• Escalate from easier sub-problems first, then feed those results into the harder target.
• Add an independent verification layer — Lean formalization, expert review, or a separate checker model.

That last step is non-negotiable. OpenAI spent an additional 17 hours formalizing its proof in Lean (per its own report), and ProofCouncil had one solution externally verified by three independent experts (the evaluation notes). Watch for the top failure mode, too: agents most often solve an easier version of the problem than the one you asked — misinterpretation, not bad math.

Agent skills are the building blocks of this scaffolding, and choosing them deserves the same skepticism. SkillzBench is building a free directory where every skill is version-pinned to a commit SHA, security-scanned, and sandbox-benchmarked with a published Lift — plus a free MCP connector — so you can assemble and test your own agent loadouts with evidence instead of hype.