How Does AI Image Detection Work?

Why AI Image Detection Is Broken Right Now

In 28 minutes, for $8, one journalist fabricated an entire political scandal — complete with fake news anchors, protest footage, and a fictional mayor — and every image looked real. That experiment, documented by the Global Investigative Journalism Network, shows how cheap and fast synthetic media has become while our defenses lag far behind.

The problem: most people's mental checklist for spotting AI images is stuck in 2023. Back then, you could count fingers, look for garbled text, or check for misaligned ears. Those tells no longer work. Midjourney and DALL-E now render anatomically correct hands, and OpenAI trained DALL-E 3 specifically on text accuracy, so the "obvious" giveaways have vanished.

Worse, outdated training creates a false sense of security. As verification expert Henk van Ess warns, a journalist trained on 2023 methods might declare obvious AI content authentic simply because it passes obsolete tests — and that misplaced certainty is more dangerous than honest uncertainty.

The technical defenses have cracks too. The three pillars of AI image detection — watermarking, provenance metadata, and algorithmic analysis — each fail in different ways:

• Metadata can be stripped: C2PA "Content Credentials" are attached to files rather than embedded in them, so a malicious actor can remove them with a simple online tool, according to Digimarc.
• Watermarks are deterrents, not guarantees: Google DeepMind admits SynthID is not designed to directly stop motivated adversaries.
• Detectors don't generalize: a peer-reviewed review of 2018–2025 research found detection systems lose 10–15% accuracy when tested across datasets, suggesting they learn dataset-specific compression artifacts rather than true deepfake characteristics (MDPI).

For businesses adopting AI, the stakes are practical, not abstract. If you rely on AI-generated visuals in marketing or operations, you also inherit the verification problem: a competitor, a fraudster, or a regulator can ask whether your imagery is authentic — and "it looked fine" is not an answer. Regulation is tightening, too: China's GB 45438-2025 standard now mandates labeling and detection features for AI-generated content, and the EU AI Act fully applies from August 2026, pushing provenance and detection tooling into commercial pipelines.

The honest takeaway is that no single method — metadata, watermark, or algorithm — is sufficient on its own. Detection has shifted from definitive identification to probability assessment, and anyone verifying images needs layered checks rather than one trusted tool. For teams building AI workflows, verification steps like metadata checks and detection API calls can be automated with agent skills; SkillzBench is building a free directory and MCP connector to help you find and test skills for exactly these kinds of pipelines.

The Three Pillars: Watermarks, Metadata, and Detection Algorithms

No single technique can tell you whether an image was made by AI. Detection rests on three different pillars — provenance metadata, invisible watermarks, and algorithmic analysis — and each one works well until someone deliberately tries to defeat it.

Pillar one: provenance metadata. The C2PA standard (Coalition for Content Provenance and Authenticity) attaches cryptographically signed "Content Credentials" to digital files, functioning like a nutrition label that records where an image came from and how it was edited. Cameras from Leica and Nikon, editing tools like Photoshop, and AI generators including Adobe Firefly and OpenAI's DALL-E all attach these manifests. The catch, as Digimarc's innovation VP Dominique Guinard explains, is that manifests are attached as metadata rather than embedded in the pixels — a malicious actor can strip them with a simple online tool. Missing metadata, then, is not proof of authenticity.

Pillar two: watermarks. Because metadata can be detached, the industry is layering digital watermarks underneath it. Google DeepMind's SynthID embeds an imperceptible watermark into AI-generated images at the moment of creation, designed to survive cropping, filters, and lossy compression. Its detector then scores arbitrary content and returns one of three states: watermarked, not watermarked, or uncertain. Even Google is candid about the limits — the watermark is "not designed to directly stop motivated adversaries," making it a deterrent rather than a guarantee. C2PA version 2.0 now includes a standard action to bind a watermark to a manifest, so a stripped file can still be recovered.

Pillar three: algorithmic detection. When neither metadata nor a watermark exists, detectors analyze the image itself. AI generators leave mathematical signatures in pixel arrangement, compression patterns, and — most importantly — the frequency domain, where artifacts appear as fingerprints invisible to the human eye. Practical geometric checks still help too: vanishing point analysis, shadow consistency, and reflection geometry, which academic GradCam studies confirmed as reliable signals.

But this pillar has a documented weakness. A peer-reviewed systematic review of detection methods from 2018 to 2025 found performance drops 10–15% on average when models are tested across datasets — evidence that detectors largely learn dataset-specific compression artifacts rather than generalizable deepfake characteristics. Transformer-based architectures fare somewhat better (an 11.33% decline versus more than 15% for CNNs) but require 3–5 times more computing power.

The practical takeaway for anyone verifying images:

• Check Content Credentials metadata first — but treat its absence as unknown, not authentic.
• Run watermark detection where the tooling exists, accepting "uncertain" as a real answer.
• Use geometric checks (shadows, reflections, vanishing points) instead of obsolete tells like malformed hands, which no longer work on 2025-era models.
• Treat every result as a probability, never a verdict.

Layering matters because each pillar fails differently. And since verification involves repetitive steps — checking manifests, calling detection APIs, logging results — it's exactly the kind of workflow agent skills can automate; SkillzBench is building a free directory where you can browse and benchmark skills for tools like Claude Code, Codex, and Cursor before trusting them with the job.

Why No Single Method Wins the Arms Race

Ask three different tools whether the same image is AI-generated and you may get three different answers — and all three could be wrong. That's not a bug in any one detector; it's the predictable result of how these systems are built.

A systematic review of detection methods from 2018 to 2025 found that performance drops by 10–15% on average when models are tested on datasets they weren't trained on. The conclusion is uncomfortable: current detectors largely learn dataset-specific compression artifacts rather than generalizable deepfake characteristics. In other words, a detector trained on one generator's output often fails when a new model arrives with different fingerprints.

Transformer-based architectures fare somewhat better, with an 11.33% cross-dataset decline versus more than 15% for CNNs — but they require 3–5 times more computation to run. That tradeoff matters for any platform scanning images at scale. Interestingly, the same review found a Random Forest classifier hit 99.64% accuracy on the DFDC benchmark with dramatically lower compute, challenging the assumption that deep learning always wins.

Even watermarking — the strongest anchor in the stack — comes with caveats. Google DeepMind's own SynthID documentation states plainly that it is "not designed to directly stop motivated adversaries," though it can make malicious use of AI content harder. Its detector outputs three states — watermarked, not watermarked, or uncertain — reflecting that detection has shifted from definitive identification to probability assessment. The GIJN practitioner guide reaches the same verdict: the goal is no longer a verdict but "informed editorial judgment," and journalist Henk van Ess warns that outdated 2023-era checks (malformed hands, garbled text) now create dangerous false confidence.

The practical takeaway for businesses is a layered approach:

• Check C2PA Content Credentials first — but remember manifests can be stripped, so absence of metadata proves nothing.
• Run watermark detection where available, treating results as likelihood scores, not verdicts.
• Fall back on durable geometric checks: shadow consistency, vanishing points, and reflection geometry.
• Communicate uncertainty honestly rather than declaring images "real" or "fake."

Regulation is now turning this layering from best practice into obligation. China's GB 45438-2025 standard, implemented September 2025, mandates labeling and detection features for AI-generated content, while the EU AI Act fully applies from August 2026 — pushing synthetic-media detection and provenance tooling into commercial image pipelines whether vendors like it or not.

If your team is building verification workflows around these tools, agent skills can automate pieces of it — checking metadata, routing images through detection APIs. SkillzBench is building a free, independently benchmarked directory of such skills, with a free MCP connector, so you can see which ones actually lift performance before you rely on them.

A Practical Layered Workflow for Verifying Images

No single test can tell you whether an image is AI-generated — but a layered workflow gets you far closer to the truth than any one tool. Investigators who built an entire fake political scandal in 28 minutes for $8 showed how fast synthetic imagery now moves, so your verification process needs to be both structured and fast.

Start with provenance. The C2PA standard's "Content Credentials" work like a nutrition label for digital content, recording where an image came from and how it was edited — and they're now attached by cameras like Leica and Nikon, editing tools like Photoshop, and generators like Adobe Firefly and DALL-E, according to the C2PA coalition. But here's the critical caveat: manifests are attached as metadata, not embedded in the image itself, so a malicious actor can strip them with a simple online tool. Missing metadata is never proof of authenticity.

Next, check for watermarks where detection is available. Google DeepMind's SynthID embeds an invisible watermark at the moment of generation, designed to survive cropping, filters, and compression, and its detector outputs three states — watermarked, not watermarked, or uncertain — reflecting how probabilistic these tools are by design. Even Google notes watermarks aren't built to stop motivated adversaries.

Then apply the durable geometric checks that still work in 2025, unlike the obsolete "count the fingers" tests:

• Shadow consistency — all shadows should point away from a single light source; AI images often show varied shadow directions.
• Vanishing point analysis — academic GradCam studies found structural distortions near vanishing points in AI-generated scenes.
• Reflection geometry — mirrors and water reflections should match the physics of the scene.

Finally, treat every result as a probability score, not a verdict. Detection has shifted "from definitive identification to a probability assessment," and a peer-reviewed systematic review found detectors lose 10–15% of their accuracy when tested across datasets — they often learn dataset-specific compression artifacts rather than genuine deepfake characteristics. A journalist trained on outdated 2023 tells can develop false confidence, which is more dangerous than honest uncertainty.

This workflow has several repeatable, mechanical steps — reading C2PA manifests, calling watermark and detection APIs, logging probability scores — and that's exactly the kind of work agent skills can automate. SkillzBench is building a free directory of vetted agent skills plus a free MCP connector, so you can find skills that handle metadata checks and detection API calls, see how each one is security-scanned and benchmarked, and wire them into your own verification pipeline. Join the waitlist to get access as it launches.

One caution from practitioners: don't try to build a detector yourself. Without massive computing power and a Ph.D.-level team, you'll fail — layering existing tools is the practical path.