Do AI Detectors Work? Accuracy, Bias and Turnitin

Support

Quick answer: AI detectors work sometimes, but not reliably enough to prove anything on their own. They can flag text that looks statistically like chatbot output, yet they also mislabel genuine human writing and can be fooled by light editing. OpenAI withdrew its own AI text classifier in July 2023 because of its “low rate of accuracy”, after it correctly caught only 26% of AI-written text in tests. A Stanford-led study found seven popular detectors labelled 61% of essays by non-native English speakers as AI-generated. Treat a detector score as a reason for a conversation, not as evidence.

Since ChatGPT arrived in late 2022, AI detectors have become a fixture in classrooms, newsrooms and hiring. Teachers paste essays into them, students panic over them, and editors argue about them. The uncomfortable truth is that even the companies building AI have struggled to make detection dependable. If you have been flagged, or you are deciding whether to rely on one, here is what the evidence says.

In This Article

How Do AI Detectors Work?

Most AI detectors are themselves machine-learning models trained on examples of human and AI text. They look for statistical fingerprints: how predictable each next word is (sometimes called “perplexity”) and how much sentence length and structure vary (“burstiness”). Chatbot text tends to be smooth, predictable and evenly paced, while human writing is usually lumpier. The detector then outputs a probability or a percentage of text it thinks is AI-generated.

The problem is that plenty of humans write predictably too, especially in formal essays, technical reports, or a second language, and AI text can be made less predictable with a few edits or a different prompt.

Are AI Detectors Accurate?

The evidence is not encouraging:

OpenAI’s own tool failed. In January 2023 OpenAI launched a classifier and said it correctly identified 26% of AI-written text as “likely AI-written” while wrongly labelling human-written text as AI 9% of the time. It said the tool was “very unreliable” on texts under 1,000 characters. On 20 July 2023, OpenAI withdrew it, citing its low rate of accuracy.
Non-native English writers are flagged far more often. In a 2023 study published in the journal *Patterns*, researchers led by Stanford’s James Zou tested seven widely used detectors. They were near-perfect on essays by US eighth-graders but classified 61.22% of TOEFL essays written by non-native English speakers as AI-generated, according to Stanford HAI. All seven detectors agreed on 19% of those human-written essays being AI.
Simple edits fool them. In the same study, detectors initially spotted ChatGPT-written college admission essays, but when ChatGPT was asked to rewrite them with more literary language, detection rates plummeted to near zero.
Even a small error rate adds up. Vanderbilt University noted that Turnitin claimed a 1% false positive rate at launch. With 75,000 papers submitted in 2022, that could have meant around 750 students wrongly flagged. Vanderbilt disabled Turnitin’s AI detector in August 2023.

Does Turnitin’s AI Detector Work?

Turnitin, already the standard plagiarism checker at many schools and colleges, added AI writing detection in April 2023, which made it one of the most widely encountered detectors. Some universities use its AI score as one signal among many. Others, including Vanderbilt, have switched it off, citing concerns over false positives, bias against non-native speakers and a lack of transparency about how it works. Policies vary, so check your institution’s academic integrity guidance.

Why Do AI Detectors Flag Human Writing?

Formulaic writing. Five-paragraph essays, lab reports and legal or technical writing follow predictable patterns.
Writing in a second language. Learners often use simpler, more common vocabulary, which looks “predictable”.
Grammar and editing tools. Heavy use of tools that rewrite sentences can smooth out human quirks.
Short passages. With little text, there is not enough signal, as OpenAI acknowledged.

What Should You Do If an AI Detector Flags Your Work?

Stay calm and ask what evidence is being used. A detector score alone is weak evidence.
Show your process. Version history in Google Docs or Microsoft Word, notes, outlines, drafts and research sources are powerful proof of authorship.
Offer to discuss the work. Explaining your argument and sources in person demonstrates understanding.
Check the policy. Many schools have rules requiring more than a detector score for misconduct findings.
Ask for a review or appeal through your institution’s formal process if needed.

How Can Teachers and Employers Check for AI Use Fairly?

Set clear rules on what AI use is allowed, and when it must be disclosed.
Design assignments that require personal reflection, drafts, in-class work or oral follow-up.
Use detector results only as a prompt for a conversation, never as standalone proof.
Look for other signals, such as invented citations, generic content or a sudden change in style compared with earlier work.
Be especially careful with non-native English speakers, given the documented bias.

Will AI Detection Get Better?

Researchers are exploring “watermarking”, hidden statistical patterns built into AI output when it is generated, and content provenance standards that record how a piece of content was made. These approaches depend on AI companies adopting them and can still be weakened by paraphrasing. For now, most experts agree that no tool can reliably determine whether an individual piece of text was written by a human or an AI. If you want to understand where AI tends to go wrong, our explainer on how retrieval-augmented generation works and where it fails is a good start.

Frequently Asked Questions

Can AI detectors be wrong?

Yes. They produce both false positives (human text flagged as AI) and false negatives (AI text passed as human). OpenAI’s withdrawn classifier wrongly flagged human text 9% of the time in its own tests.

Do AI detectors work on ChatGPT?

They can catch unedited ChatGPT text some of the time, but lightly edited or paraphrased output often slips through.

Do AI detectors work on code?

Poorly. OpenAI said its classifier was unreliable on code, and code is often predictable whoever writes it.

Can a teacher prove I used ChatGPT?

Not from a detector score alone. Stronger evidence includes invented references, inconsistencies with your earlier work, or an inability to explain what you submitted.

What is the most accurate AI detector?

There is no independently verified “most accurate” detector. Accuracy varies by text type, length and language, and all have documented errors.

The Bottom Line

AI detectors are, at best, rough indicators. They miss plenty of AI text and wrongly accuse some humans, particularly non-native English writers. Use them carefully, keep your drafts and version history, and focus on transparency about how AI was used. For practical skills to work alongside AI rather than against it, read AI and jobs: practical skills for working with generative AI.

Sources

OpenAI — New AI classifier for indicating AI-written text (updated 20 July 2023) (31 January 2023).
Patterns (Liang et al.) — GPT detectors are biased against non-native English writers (July 2023).
Stanford HAI — AI-Detectors Biased Against Non-Native English Writers.
Vanderbilt University — Guidance on AI Detection and Why We’re Disabling Turnitin’s AI Detector (16 August 2023).