Can Agent Skills Handle Wikipedia Signs of AI Writing? What to Expect in 2026

Why Wikipedia's AI-Writing Problem Is Now a Policy Crisis

In March 2026, Wikipedia moderators voted 40-to-2 to ban LLM-generated article content across a platform serving roughly 260,000 editors. The vote, reported by mainstream coverage, marked the moment AI writing stopped being a nuisance and became a formal policy crisis for the world's largest reference work.

The scale of the problem explains the severity. A Princeton-affiliated study found that automated detectors flagged over 5% of newly created English Wikipedia articles as AI-generated, with thresholds calibrated to a 1% false positive rate. Flagged articles were typically lower quality and often self-promotional.

Wikipedia didn't rely on policy alone. Through 2025 and 2026, it deployed a layered detection stack:

• An Abuse Filter analyzing edits for possible AI-generated text, live December 5, 2025
• A follow-up Abuse Filter targeting AI-sourced citations, deployed September 16, 2026
• An LLM Paste Check tool, default-on for English Wikipedia from September 24, 2026, that warns when pasted text carries metadata from major generative AI chatbots
• Human review through WikiProject AI Cleanup, the volunteer group founded in 2023 that handles informal detection and enforcement

This hybrid model — software, wiki-scripts, and humans — reflects an uncomfortable truth documented across the research: no single detector is trustworthy. An evaluation of 14 detection tools including Turnitin and GPTZero found all scored below 80% accuracy, and one study measured a 61.3% average false positive rate against non-native English writers. Wikipedia's own guidance explicitly warns that detector scores are not valid grounds for deletion.

The crisis matters far beyond Wikipedia. Ilyas Lebleu, co-founder of WikiProject AI Cleanup, described AI's ability to "mass-produce content that sounds real while being completely fake" — and warned of a domino effect as other platforms decide whether AI is welcome on their own terms. The worst documented failures weren't stylistic at all: one AI-written historical draft included seven references, five of them fabricated.

That's the real lesson for anyone publishing, editing, or moderating web content in 2026. Surface tells — canned phrases, Markdown artifacts, overused em dashes — are flaggable, and agent skills can encode Wikipedia's structured Signs of AI Writing taxonomy into automated triage. But attribution is a different problem entirely, and the stakes run to fabricated citations, not awkward prose.

This is also why skill quality matters. With published research finding roughly 26% of agent skills carrying at least one vulnerability, SkillzBench is building version-pinned verification and sandbox benchmarking so teams deploying detection skills know exactly what they're running — and what lift each skill actually delivers.

The Signs of AI Writing: A Taxonomy Skills Can Actually Encode

Most AI-detection conversations devolve into vague vibes. Wikipedia did the opposite: its community turned AI writing tells into a structured, pattern-based guide that reads almost like a specification. That structure matters, because a list of documented patterns is exactly the raw material an agent skill can encode and act on.

The Signs of AI Writing guide, maintained by WikiProject AI Cleanup, catalogs the most common tells in newly AI-generated text. Some are stylistic: canned phrases like "stands as a testament" and "pivotal moment," hedging disclaimers about source availability, excessive em dashes, and overuse of "moreover." Others are almost purely mechanical.

That mechanical layer is where agent skills shine. Wikipedia's own detection stack already leans on machine-checkable signals, including an LLM Paste Check tool that warns when pasted text carries metadata from major generative chatbots, plus Abuse Filters deployed in late 2025 and 2026. Technical signals are simply easier to verify than tone:

• Markdown artifacts leaking into wiki text
• AI-style edit summaries and LLM-response phrasing like "Here is your Wikipedia article on"
• Curly versus straight quotation marks and unusual Unicode or space characters
• Chatbot metadata embedded in pasted content

Stylistic signals deserve lower confidence. The guide itself warns that detectors like GPTZero and Pangram "have non-trivial error rates," and a 2025 study found human recognition of AI text is no better than random chance. Hacker News commenters add a sharper critique: publishing detection lists invites adversarial bypass, since prompters simply instruct models to avoid flagged phrases — though even then, LLMs remain "still highly likely to write like this and hallucinate information."

The taxonomy's real value is as a triage layer, not a verdict. Wikipedia explicitly cautions against treating the signs as problems to fix in isolation, because they indicate deeper policy violations — most dangerously, fabricated references. One AI-written historical draft cited seven sources, five of which were fake. That's why the guide frames surface tells as entry points for investigation rather than proof of authorship.

For anyone building a Wikipedia-scanning skill in 2026, the playbook is clear: encode the machine-checkable patterns first, weight phrase-matching lower, and calibrate thresholds the way the Princeton-affiliated preprint did — targeting a 1% false positive rate on known-human text. Expect to flag roughly 5% of new articles as candidates, and route every flag to human review or citation verification rather than issuing a verdict.

Because these signal lists decay as models adapt, skills built on WP:AISIGNS need versioning and periodic re-testing against fresh baselines — which is precisely why SkillzBench is building version pinning and sandbox benchmarking into its free skills directory, so a detection skill's real-world lift stays measurable as the underlying models change.

Why Flagging Works but Proving Fails: The Detection Accuracy Problem

If you feed a Wikipedia article into an AI detector and get a high score, that number proves almost nothing. The research on detector reliability is brutal, and it explains why even Wikipedia's own editors treat detection scores as a starting point for suspicion, never as evidence.

The academic record is unambiguous. A study evaluating 14 detection tools, including Turnitin and GPTZero, found that all scored below 80% accuracy, with only five clearing 70%, according to Wikipedia's overview of AI content detection. The same research summary documents a 61.3% average false positive rate for non-native English writers across seven GPT detectors — meaning the tools disproportionately flag humans who write in a second language.

Adversarial manipulation makes things worse. One August 2023 study found Originality.AI detected GPT-4 text at 91.3% mean accuracy, but after simple adversarial reprocessing of the text, performance collapsed to 27.8%. And humans don't fare much better: a 2025 study found people distinguish LLM text at roughly random chance, though a smaller preprint reported expert annotators reaching 90% accuracy with just 3.3% false positives.

Wikipedia has internalized these limits. Its Signs of AI Writing guide explicitly warns that detectors like GPTZero and Pangram "have non-trivial error rates," and states that a high detector score is not valid grounds for speedy deletion under G15. Janelle Shane's advice — "don't use AI detectors for anything important" — carries weight because portions of her own human-written book were flagged as AI.

Three structural problems guarantee this won't be solved by better phrase lists:

• Signal decay: human writing is increasingly influenced by LLMs (detectable in spoken content as early as 2024), so human and AI styles converge even as the detection lists age.
• Adversarial bypass: publishing detection lists lets prompters simply instruct models to avoid flagged phrases — though commenters note LLMs are "still highly likely to write like this" anyway.
• The wrong target: style-based detection punishes bad human writing too, since weasel words and puffery are policy violations regardless of authorship.

This is exactly why agent skills built for Wikipedia scanning should output "flag for review," never "confirmed AI" — and why versioned, re-benchmarked skills matter, since stale pattern lists degrade as models adapt. That verify-and-benchmark discipline is what SkillzBench is building into its free skills directory: pinning every skill to a version, testing it in a sandbox, and publishing the measured Lift so users can see what a skill actually contributes. Flagging is achievable. Proof is not.

Building a Triage Skill: Calibrated Flags, Fabricated-Reference Escalation

The blueprint for a Wikipedia-scanning agent skill starts with a design decision: it should behave like a smoke detector, not a judge. The evidence is unambiguous that detectors cannot prove authorship — a Wikipedia's own guidance explicitly warns that tools like GPTZero and Pangram "have non-trivial error rates" and that a high detector score is not valid grounds for deletion. So the skill's output vocabulary should contain exactly one verdict: flag for review, never "confirmed AI."

Calibration is where most builds fail. The strongest available benchmark comes from an arXiv preprint that flagged over 5% of newly created English Wikipedia articles using GPTZero and Binoculars — but only after tuning thresholds against pre-GPT-3.5 articles to hold the false positive rate at 1%. That methodology is replicable: run your skill against a known-human baseline corpus and adjust until it flags almost nothing that is verifiably human. Skip this step and you inherit the industry's failure modes — across 14 evaluated tools, all scored below 80% accuracy, and one study found a 61.3% average false positive rate against non-native English writers.

Signal prioritization matters just as much. The most machine-checkable tells are technical, not stylistic:

• Generative chatbot metadata embedded in pasted text — the basis of Wikipedia's own LLM Paste Check tool
• Markdown artifacts like bolded lead sentences and heading syntax
• AI-style edit summaries and LLM-response phrasing such as "Here is your Wikipedia article on"
• Unusual Unicode, special space characters, and em dash overuse

Phrase-matching against canned expressions like "stands as a testament" deserves a lower weight, because publishing detection lists invites adversarial bypass — prompters simply instruct models to avoid flagged phrases. Notably, Hacker News commenters observe that LLMs are "still highly likely to write like this and hallucinate information" even when told not to, which is why the escalation path matters more than the phrase list.

That escalation path should target fabricated references. Wikipedia's guide warns against treating surface signs as the problems themselves — they indicate deeper policy violations, and the worst documented failures are citation frauds: one AI-written draft included seven references, five of them fabricated, and another was a 2,000-word article about a nonexistent Ottoman fortress with elaborate fake citations. A skill that cross-checks flagged citations against real sources delivers far more value than any style heuristic.

The architecture mirrors Wikipedia's own hybrid model of software, wiki-scripts, and human review — and skills deployed in this role need versioning and re-testing as models evolve, since signals decay. That is precisely the gap SkillzBench is being built to address: pinning skill versions, sandbox-benchmarking them, and publishing Lift scores so builders can see whether a triage skill actually helps before wiring it into an agent.

What to Expect in 2026 and How to Test Skills Before You Trust Them

If 2025 was the year Wikipedia panicked about AI writing, 2026 is the year it institutionalized the response — and the numbers suggest the problem won't shrink. Automated detectors using GPTZero and Binoculars flagged over 5% of newly created English Wikipedia articles as AI-generated, with thresholds calibrated to a 1% false positive rate, according to a Princeton-affiliated preprint. Treat that 5% as a realistic floor for what any scanning skill should surface.

Expect three things to define the year ahead:

• Signal decay accelerates. Phrase lists like "stands as a testament" and "pivotal moment" age quickly, since prompting an LLM to avoid flagged phrases is trivial — though one community discussion counters that models "are still highly likely to write like this and hallucinate information" even when told not to.
• Machine-checkable signals win. Wikipedia's own stack leans on technical tells — chatbot metadata in pasted text, Markdown artifacts, unusual Unicode — via its Abuse Filter and default-on LLM Paste Check tool deployed across language editions in 2026.
• Wikipedia's governance model spreads. The platform's March 2026 ban on LLM-generated article content passed 40-to-2 among moderators, and researchers call its approach "minimalist constitutionalism" — narrow rules after comprehensive proposals failed twice. WikiProject AI Cleanup's Ilyas Lebleu predicts a "domino effect" as other platforms make the same call.

For anyone testing an agent skill against these signals before trusting it, the bar is calibration, not vibes. The replicable benchmark comes from the academic method itself: measure flagged rates against a known-human baseline at a fixed false positive rate, and re-test as models change. Because detection lists decay, a skill built in January may misfire by June — which is why versioning and repeat testing matter more than a single impressive demo.

That's the gap SkillzBench is being built to close. Its approach pins every skill to a specific version, runs sandbox benchmarks with and without the skill under identical conditions, publishes the Lift score separately from ratings, and re-scans when upstream skills change — with security scanning included, given published research found roughly 26% of skills carry at least one vulnerability.

The practical takeaway: deploy skills as a triage layer, not a verdict machine. Wikipedia's own guide warns that detectors "have non-trivial error rates" and that a high detector score is not grounds for deletion — the real target is deeper failures like fabricated references, where one AI-written draft included seven citations, five of them fake. Flag for review, escalate to humans, and verify the sources. When you're ready to try it yourself, you can join SkillzBench's waitlist to browse and test free agent skills built on exactly these principles.