Is Grammarly actually good at detecting AI?

The Contradictory Evidence: Grammarly's Split Test Results

Two tests, two opposite verdicts. One peer-reviewed study chose Grammarly as one of the few detectors that passed its pilot; an independent-style benchmark gave it a perfect zero on AI text. Both can't be dismissed, and the gap between them tells you most of what you need to know.

The positive signal comes from a 2025 peer-reviewed study by Cheng et al., published in Advances in Simulation. The researchers ran pilot testing to see which tools could reliably differentiate 100% human-written versus 100% AI-written content. Grammarly passed and earned a spot in the study — while GPTZero was excluded for frequently categorizing human text as AI-written. That's a meaningful endorsement: in controlled, clean conditions, Grammarly separated the extremes.

Then comes the contradiction. In Pangram Labs' benchmark of 30 AI detectors, Grammarly passed 0 of 9 tests on AI-generated text — a 0% detection rate — while correctly identifying 3 of 3 human-written texts. Pangram's blunt takeaway: Grammarly "didn't pass any of our tests for detecting AI generated content accurately." The pattern is a striking false-negative profile: the tool rarely cries wolf, but it also misses the wolf almost every time.

Both results deserve scrutiny, and neither is a clean win:

• The peer-reviewed study tested only the extreme binary case — pure human versus pure AI — not the mixed, lightly edited text people actually submit.
• Pangram is a competitor vendor whose own detector conveniently scored 100% in its own benchmark.
• The study itself found cross-tool reliability ranging from ICC 0.57 to 0.95, which the authors said raised "concerns regarding overall reliability."

There's also a broader lesson here about trusting tool claims. Pangram itself notes that many detectors advertise "99% accuracy" that collapses under independent testing — exactly the gap between marketing and reality that motivates SkillzBench's approach of verifying AI tool performance in standardized, sandboxed benchmarks rather than taking claims at face value.

So what's the honest verdict on the contradictory evidence? Grammarly appears strong on extremes but weak in practice. It can sort purely human text from purely AI text in a lab setting, yet it failed to flag AI-generated content in Pangram's more realistic testing. For anyone evaluating Grammarly's detector, that split result is the answer: useful as a signal, unreliable as a verdict — and the peer-reviewed authors themselves concluded these tools "should not be relied upon alone at this time."

Why the Entire AI Detection Category Is Broken

If you think Grammarly's detector is the weak link in AI detection, the uncomfortable truth is that the entire category is struggling. Even the people building these tools can't agree on what "accurate" means, and some of the world's best-funded institutions have quietly concluded the problem may be unsolvable.

The institutional retreat is the clearest signal. MIT Sloan's official guidance warns that detection software has high error rates and can lead instructors to falsely accuse students, noting that OpenAI shut down its own AI detector over poor accuracy. Vanderbilt, Yale, Johns Hopkins, Northwestern, UCLA, and at least seven other elite universities disabled Turnitin's AI detector outright, and UT Austin banned purchasing detection tools in 2024, according to reporting gathered by industry comparisons.

The bias data explains why. Across seven detectors, 61% of TOEFL essays written by non-native English speakers were wrongly flagged as AI-generated, compared to roughly 5% for native speakers, per MIT Sloan's analysis. Stanford researchers found detectors misclassified an average of 61.3% of those essays, with 97.8% flagged by at least one tool. A detector that punishes writers for writing carefully is not a detector — it's a liability.

Evasion is just as damning. Simple prompt editing dropped detection of genuinely AI-written college essays from 100% to 13%, and paraphrasing cut detection rates by roughly 85% on average. University of Maryland professor Soheil Feizi puts it bluntly: after paraphrasing, detectors perform "little better than a random guess," and as language models improve, reliable detection will get even harder.

Even humans can't do this job. In a peer-reviewed 2025 study, human raters identified AI-written text with only 19% accuracy — indistinguishable from chance. That finding, published in the study itself, reframes the whole debate: if people score at coin-flip level, any detector claiming near-perfection deserves scrutiny, not trust.

And scrutiny is warranted. Pangram Labs notes that many detectors advertise "99% accuracy" but fall far short in real testing, citing the RAID benchmark's finding that most detectors fail when forced to keep false positives under 1%. Independent academic testing has reached similar conclusions — one study of 14 detectors found them "neither accurate nor reliable," with most scoring below 80% accuracy.

• 12+ elite universities disabled Turnitin's AI detector; UT Austin banned purchasing detection tools entirely.
• 61% of TOEFL essays by non-native English writers were falsely flagged as AI-generated.
• Simple paraphrasing reduces most detectors to near-random performance.
• OpenAI shut down its own detector because it couldn't get accuracy right.

This is exactly the gap SkillzBench is built to address: independent, sandboxed benchmarking that publishes results separately from marketing claims, so users can see verified performance rather than vendor promises. Until standardized verification becomes the norm, treat every accuracy figure — including Grammarly's — as an unproven claim.

The Real Stakes: False Accusations and Career-Altering Mistakes

A single false accusation can end a career. That's the reality behind the accuracy numbers: when an AI detector flags a student, a researcher, or a professional writer, the human cost falls on someone who may have done nothing wrong.

The peer-reviewed literature is blunt about this. A study in Acta Neurochirurgica warned that it is "not uncommon" for detectors to wrongfully analyze genuinely human-written text, with "far-reaching consequences, such as the possibility of career-altering accusations against researchers whose work is, in fact, original." These aren't hypothetical harms — they're documented risks in one of the most rigorous fields in academia.

Even the best-known enterprise tools admit the problem. Turnitin's chief product officer acknowledged that "we cannot mitigate the risk of false positives completely," and the company's own sentence-level false flag rate stands at 4%. In a large class or a long manuscript, that error rate compounds quickly.

The math of false positives is unforgiving. University of Maryland researcher Soheil Feizi argues that any detector used for high-stakes decisions would need a false-positive baseline of just 0.01% — a threshold that is currently "impossible" to achieve. The RAID benchmark, which tested 11 detectors across 6.28 million texts, found most detectors fail outright when forced to operate below a 1% false-positive rate.

Reliability varies wildly depending on which tool you ask. In the peer-reviewed simulation study that tested Grammarly directly, cross-tool agreement ranged from ICC 0.57 to 0.95 — a spread the authors said raised "concerns regarding overall reliability of these tools." In practice, that means two detectors can look at the same document and disagree, and the outcome may hinge on which one an institution happens to use.

The bias problem makes it worse:

• 61% of TOEFL essays by non-native English speakers were wrongly flagged as AI-generated across seven detectors, versus roughly 5% for native speakers.
• Simple prompt editing dropped detection of genuinely AI-written essays from 100% to 13%, meaning evasion is trivial while false accusations persist.
• Paraphrasing reduces detector performance to "little better than a random guess," according to Feizi's research.

This is exactly why the peer-reviewed authors concluded that detection tools "should not be relied upon alone at this time" — a caution that applies to Grammarly's detector as much as any other. It's also the gap SkillzBench is built to address: independent, sandboxed benchmarking that verifies what tool marketing claims, rather than taking "99% accuracy" at face value. Until independent verification is the norm, no detector score — Grammarly's included — belongs anywhere near a high-stakes decision.

What to Do Instead: A Smarter Approach to AI Content Verification

If you can't trust a detector's verdict, what can you trust? The answer emerging from researchers and institutions is refreshingly practical: treat AI detection as one weak signal, and build your process around transparency instead of prediction.

The peer-reviewed evidence points the same way. Cheng et al.'s 2025 study concluded that while detection tools "may assist in determining potentially unethical use," they should not be relied upon alone at this time — cross-tool agreement between detectors was so variable (ICC 0.57–0.95) that two tools could disagree about the same document.

MIT Sloan has gone further, recommending that educators move away from detection entirely. Their guidance favors clear AI-use policies, transparency, and process statements over automated verdicts — an approach that assumes AI use is likely and asks people to disclose it, rather than trying to catch them (MIT Sloan).

In practice, that means a few habits:

• Never treat any detector score as sole evidence for an accusation — the stakes are too high, with researchers warning of "career-altering accusations" against authors whose work is genuinely original.
• Ask writers for a short process statement: what tools they used, where, and how they edited the output.
• Use detectors, if at all, as a prompt for a conversation — not a verdict.
• Evaluate writing quality and source accuracy directly, since human raters managed only 19% accuracy at spotting AI text — essentially chance.

It also means reading vendor marketing with a skeptical eye. Pangram Labs notes that many detectors claim "99% accuracy" but fall far short in real testing, and the RAID benchmark found most detectors fail outright when forced to keep false positives under 1%. Grammarly's own results illustrate the gap between pitch and performance: it passed 0 of 9 AI-generated text tests in one vendor benchmark, despite its polished reputation.

This is exactly the problem SkillzBench is being built to address. AI tool claims deserve independent, standardized verification — the same sandboxed testing, transparent scoring, and separately published performance results that SkillzBench is applying to AI agent skills. Until that kind of verification becomes the norm, the smartest move with any AI detector — Grammarly included — is the same: use it as one input, verify independently, and never let a percentage score make a high-stakes decision for you.

Why Independent, Standardized Testing Matters — The SkillzBench Approach

The same detector, the same year, two very different report cards. A peer-reviewed study selected Grammarly's AI detector because it reliably distinguished 100% human from 100% AI text in pilot testing, while GPTZero failed that basic bar. Yet an independent-style benchmark of 30 detectors found Grammarly passed 0 of 9 tests on AI-generated text — a complete miss — while correctly identifying 3 of 3 human-written samples.

That contradiction is the whole problem in miniature. When a tool can look near-perfect in one evaluation and score zero in another, the evaluation method matters more than the marketing claim. And marketing claims are exactly where the trouble starts: as one industry analysis notes, many detectors advertise "99% accuracy" that collapses under actual testing, and the RAID benchmark found most detectors fail entirely when forced below a 1% false-positive rate. GPTZero's claimed accuracy swung from 99% to 52% depending on who ran the test.

The stakes of unverified claims are not abstract. MIT Sloan's teaching guidance warns that detectors carry high error rates and flag non-native English writers disproportionately — 61% of TOEFL essays were wrongly labeled AI-generated versus roughly 5% for native writers. One medical journal study cautioned that false flags can lead to "career-altering accusations against researchers whose work is, in fact, original."

This is why standardized, independent testing needs to be the default, not the exception. Any credible evaluation framework should hold itself to a few non-negotiables:

• The same test conditions for every tool — same inputs, same model, same scoring rules
• A controlled baseline: performance with and without the tool, so Lift is measured, not asserted
• Sandboxed execution, so results reflect the tool rather than the environment
• Published scores kept strictly separate from commercial relationships — money never touches rankings

This is the gap SkillzBench was built to close. Before launch, our lab is constructing exactly this kind of harness for AI agent skills: every skill pinned to a commit SHA, security-scanned, then benchmarked in a sandbox with the same model, with and without the skill installed, so the published Lift in percentage points reflects what actually happened — not what a landing page promised. Published research has already found that roughly 26% of skills carry at least one vulnerability, which makes independent verification a practical necessity rather than a nice-to-have.

Grammarly's split verdict shows what happens without that discipline. If you want tool claims tested before you trust them, join the SkillzBench waitlist and see verified skills, proven — not promised.