Is This Picture AI?

The Coin-Toss Problem: Why Your Eyes (and the Tools) Keep Failing You

If you flip a coin to decide whether an image is AI-generated, you'll do about as well as the average human being. A preregistered study of 1,276 participants found that people identified AI-generated images correctly just 49.4% of the time — slightly worse than random guessing. Even more unsettling: prior familiarity with synthetic media made no meaningful difference to performance.

The problem isn't just human perception. Automated detectors are struggling just as hard against modern generators. In the largest detector benchmark to date — 16 methods tested against 2.6 million images from 291 generators — no universal winner emerged, and leading tools like Midjourney v7, Flux Dev, and Firefly v4 evaded most detectors with detection accuracy of just 18–30%.

The instability of these tools is perhaps the most damning finding. The benchmark's best performer achieved 75.0% mean accuracy while the worst managed only 37.5% — a 37-point gap. More troubling, a detector that ranks best on one dataset can drop roughly 20 positions on another, meaning there's no reliable "best tool" to recommend.

Commercial tools tell a similar story of inconsistency. NewsGuard's audit of five detection tools found they disagreed on 35 of 45 identical images, with some tools misclassifying authentic photos as AI-generated. A simple filter applied to a real video caused one tool to declare it "likely AI-Generated" with 96.9% probability — a false positive that was then weaponized in a political disinformation claim.

The consequences of this uncertainty are asymmetric:

• Missed fakes spread unchecked, because AI content generation takes minutes while traditional fact-checking takes hours or days.
• False positives let bad actors "dispute reality by citing a detection tool," turning the tools themselves into disinformation weapons.
• People trained on outdated detection methods — like checking for six-fingered hands or garbled text, tells that modern generators have largely fixed — may wrongly declare AI content authentic with misplaced confidence.

For business owners and managers building verification workflows, the practical conclusion is clear: single-method detection is no longer viable. The path forward involves layering multiple approaches — visual inspection, multiple detection tools, reverse image search, and metadata analysis — rather than trusting any single signal. Agent skills can automate parts of this pipeline, from reverse image searches to multi-tool cross-referencing, and SkillzBench is building a free directory of security-scanned, benchmarked skills to help assemble exactly this kind of multi-step verification workflow.

Retire the Old Tells: What No Longer Works (and What Still Does)

If you're still hunting for six-fingered hands and garbled street signs, you're working from a 2023 playbook. Today's generators — Midjourney v7, Flux, Firefly v4 — render hands and text almost flawlessly, and clinging to outdated tells creates what GIJN's Henk van Ess calls misplaced certainty, which is "more dangerous than honest uncertainty."

The evidence is blunt. By 2025, Midjourney and DALL-E produce anatomically correct hands, and OpenAI explicitly trained DALL-E 3 on text accuracy, according to the GIJN reporter's guide. Prof. Ben Y. Zhao of the University of Chicago confirms hands are no longer a reliable tell in CBC's detector testing — though he notes small logical details, like whether a button's placement makes sense, still trip models up.

So what actually works now? Physics and context. CBC's own test image, generated by Google Gemini, was identifiable by eye not through anatomy but through warped locker locks, ceiling fixtures with no visible mounting, and an implausible number of lockers.

The modern visual checklist looks like this:

• Vanishing-point failures — parallel lines that should converge (roads, hallways, shelves) simply don't.
• Inconsistent shadows — objects casting shadows in conflicting directions from a single light source.
• Reflection errors — mirrors and water that don't match what they're reflecting.
• Uncanny perfection — magazine-quality grooming in inappropriate contexts, like flawless makeup on a disaster victim.
• Logical details — hair blending into a turtleneck, or objects placed where they'd never physically belong.

Even these tells carry a caveat: humans detect AI images at just 49.4% accuracy — below chance — in a preregistered study of 1,276 participants. Familiarity with synthetic media didn't help either, with accuracy stuck between 51.1% and 51.9% regardless of prior exposure.

That's why detection has shifted, per the GIJN guide, from definitive identification to probability assessment and informed editorial judgment. Visual inspection is one signal among several — cross-checked against reverse image search, metadata, and provenance. For teams building this into a repeatable workflow, SkillzBench is developing a free directory of security-scanned, benchmarked agent skills plus a free MCP connector, so you can assemble and test verified skills for multi-step verification pipelines rather than relying on any single check.

Use Every Signal: A Multi-Method Verification Framework

If you've been treating a detector's verdict as the final word on whether an image is real, the evidence says otherwise. In a NewsGuard audit of five commercial detection tools, the tools disagreed with each other in 35 of 45 images tested — a level of inconsistency that makes any single-tool verdict unreliable.

The reason for the disagreement is structural, not incidental. Tools disagree on what "AI-generated" even means: when shown 15 heavily AI-edited images, AI or Not caught 100% while Sightengine caught just 33%. On lightly edited images, ScamAI flagged 93% while Hive and Sightengine flagged only 27%. Each tool is making a different trade-off between false positives and false negatives — and none is a universal winner, as the largest detector benchmark to date (16 methods, 291 generators, 2.6 million images) confirmed.

False positives carry real-world consequences. NewsGuard documented an authentic video that scored 96.9% "likely AI-Generated" on Hive — simply because a filter had been applied — and the result was weaponized in a political disinformation claim disputing the video's authenticity. Ironically, Hive recorded zero false positives on NewsGuard's standard authentic set; unusual lighting, contrast, and blur (common in conflict-zone imagery) are exactly what trips these systems, as ZeroGPT's CEO acknowledged.

The expert consensus is to treat detection as probability assessment, not proof. The GIJN reporter's guide frames the goal as "informed editorial judgment" rather than definitive identification. ZeroGPT CEO Rawad Baroud recommends stacking detection results with several independent checks:

• Run the image through multiple detectors and compare verdicts, never trusting one tool alone.
• Reverse image search to trace where the image first appeared and whether it has been recaptioned.
• Metadata analysis to check for editing software signatures or missing provenance data.
• Publication history and contextual reporting to confirm the image's claimed event actually happened.

This multi-method approach matters because the speed asymmetry favors bad actors: a fact-checker's careful verification takes hours, while one researcher built a complete fake political scandal in 28 minutes for $8. And as CBC's testing showed, even a tool with reasonable accuracy on large samples can be fooled by a single well-crafted image — while malicious actors iterate fakes until detectors label them real.

For teams that want to operationalize this workflow, SkillzBench is building a free directory of security-scanned, benchmarked agent skills plus a free MCP connector — so multi-step verification pipelines like reverse image search, metadata checks, and multi-tool cross-referencing can be automated with skills you've verified before equipping.

Your Step-by-Step Image Verification Workflow

Before you trust any verdict about an image, remember the speed asymmetry: one author built a complete fake political scandal in 28 minutes for $8, while traditional fact-checking takes hours or days. Your verification workflow has to be fast, ordered, and repeatable.

Start with the original. Compression, resizing, and filters cause both false positives and missed detections, and detection tools like ScamAI's "generally require high resolution and high pixel levels" to work at all. Seek out the highest-resolution version of the image before running anything.

Then run at least two detectors. In a NewsGuard audit, five commercial tools disagreed with each other in 35 of 45 images, and the worst offender flagged authentic photos as AI 40% of the time. No single tool is a verdict.

Next, work through the checklist:

• Do a reverse image search to check publication history and whether the image has appeared in other contexts.
• Check C2PA content credentials and metadata for provenance signals, where they exist.
• Inspect physics instead of anatomy: vanishing points, shadow consistency, reflections, and "uncanny perfection" like flawless makeup on a disaster victim.
• Look for small logical details — object placement, mounting hardware, button placement — which models still get wrong.

Drop the obsolete tells. Six-fingered hands and garbled text no longer work, because modern generators render both correctly.

Finally, frame your conclusion as informed judgment, not proof. A large-scale benchmark of 16 methods across 291 generators found modern tools like Midjourney v7 and Flux Dev defeat most detectors, achieving only 18–30% detection accuracy. Even the best detector topped out at 75% mean accuracy.

For teams running this pipeline regularly, agent skills can automate the repetitive steps — reverse image search, metadata extraction, multi-tool cross-referencing. SkillzBench is building a free, security-scanned directory of agent skills plus a free MCP connector, so you can browse and test verified skills for exactly this kind of multi-step verification workflow.

Automating the Check: Verification Pipelines with Agent Skills

If no single tool can be trusted, the answer isn't a better tool — it's a better process. NewsGuard's audit of five commercial detectors found they disagreed with each other in 35 of 45 identical images, and the largest detector benchmark to date showed the best detector on one dataset can drop roughly 20 positions on another. Verification at scale therefore has to be a pipeline, not a verdict.

That's the conclusion detection researchers and practitioners keep converging on. ZeroGPT's CEO recommends pairing detection results with "source verification, reverse image search, metadata analysis, publication history, and contextual reporting before reaching a conclusion." In practice, that means assembling a repeatable workflow where each step narrows uncertainty:

• Reverse image search to establish where an image first appeared and whether its provenance checks out.
• Metadata and publication-history checks to see who published it, when, and with what editing trail.
• Cross-referencing two or more detectors, since tools trade off false positives against false negatives — Hive scored zero false positives on authentic images but flagged only 33% of heavily edited ones, while AI or Not caught 100%.
• Human review of physics and context: shadow direction, vanishing points, reflections, and "uncanny perfection" like flawless makeup on a disaster victim.

For a business handling a handful of images a week, a person can run that checklist manually. But the speed asymmetry is brutal: one author built an entire fake political scandal in 28 minutes for $8, while traditional fact-checking takes hours or days. At any real volume, the multi-step pipeline needs to be automated — and that's exactly the kind of workflow agent skills are built for. A reverse image search skill, a metadata extraction skill, a multi-tool cross-referencing skill: each handles one pipeline stage while a human makes the final judgment call.

The catch is that skills vary wildly in quality and safety. Just as no detector is universally reliable, no published skill can be taken at face value — research has found roughly 26% of agent skills carry at least one vulnerability. Equipping an agent with an unvetted skill to run security-sensitive verification is its own risk.

That's the problem SkillzBench is building against: a free, independent directory of agent skills that are version-pinned, security-scanned, license-checked, and benchmarked in a sandbox — with the performance lift shown separately from trust scores — plus a free MCP connector that lets you toggle exactly which skills your agent sees. It's in development now, with the waitlist open. For teams assembling a verification pipeline, being able to browse, test, and equip verified skills — rather than installing whatever looks plausible — turns a fragile manual checklist into a dependable, auditable workflow. The human stays in the loop for the final call; the agent does the legwork.