How Do AI Detectors Work? | Methods & Reliability

AI detectors (also called AI writing detectors or AI content detectors) are tools designed to detect when a text was partially or entirely generated by artificial intelligence (AI) tools such as ChatGPT.

AI detectors may be used to detect when a piece of writing is likely to have been generated by AI. This is useful, for example, to educators who want to check that their students are doing their own writing or moderators trying to remove fake product reviews and other spam content.

However, these tools are quite new and experimental, and they’re generally considered somewhat unreliable for now. Below, we explain how they work, how reliable they really are, and how they’re being used.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

How do AI detectors work?

AI detectors are usually based on language models similar to those used in the AI writing tools they’re trying to detect. The language model essentially looks at the input and asks “Is this the sort of thing that I would have written?” If the answer is “yes”, it concludes that the text is probably AI-generated.

Specifically, the models look for two things in a text: perplexity and burstiness. The lower these two variables are, the more likely the text is to be AI-generated. But what do these unusual terms mean?

Perplexity

Perplexity is a measure of how unpredictable a text is: how likely it is to perplex (confuse) the average reader (i.e., make no sense or read unnaturally).

• AI language models aim to produce texts with low perplexity, which are more likely to make sense and read smoothly but are also more predictable.
• Human writing tends to have higher perplexity: more creative language choices, but also more typos.

Language models work by predicting what word would naturally come next in a sentence and inserting it. For example, in the sentence “I couldn’t get to sleep last …” there are more and less plausible continuations, as shown in the table below.

Low perplexity is taken as evidence that a text is AI-generated.

Burstiness

Burstiness is a measure of variation in sentence structure and length – something like perplexity, but on the level of sentences rather than words:

• A text with little variation in sentence structure and sentence length has low burstiness.
• A text with greater variation has high burstiness.

AI text tends to be less “bursty” than human text. Because language models predict the most likely word to come next, they tend to produce sentences of average length (say, 10–20 words) and with conventional structures. This is why AI writing can sometimes seem monotonous.

Low burstiness indicates that a text is likely to be AI-generated.

A potential alternative: Watermarks

OpenAI, the company behind ChatGPT, claims to be working on a “watermarking” system where text generated by the tool could be given an invisible watermark that can then be detected by another system to know for sure that a text was AI-generated.

However, this system has not been developed yet, and the details of how it might work are unknown. It’s also not clear whether the proposed watermarks will remain when the generated text is edited. So while this may be a promising method of AI detection in the future, we just don’t know yet.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

How reliable are AI detectors?

In our experience, AI detectors normally work well, especially with longer texts, but can easily fail if the AI output was prompted to be less predictable or was edited or paraphrased after being generated. And detectors can easily misidentify human-written text as AI-generated if it happens to match the criteria (low perplexity and burstiness).

Our research into the best AI detectors indicates that no tool can provide complete accuracy; the highest accuracy we found was 84% in a premium tool or 68% in the best free tool.

These tools give a useful indication of how likely it is that a text was AI-generated, but we advise against treating them as evidence on their own. As language models continue to develop, it’s likely that detection tools will always have to race to keep up with them.

Even the more confident providers usually admit that their tools can’t be used as definitive evidence that a text is AI-generated, and universities so far don’t put much faith in them.

AI detectors vs. plagiarism checkers

AI detectors and plagiarism checkers may both be used by universities to discourage academic dishonesty, but they differ in terms of how they work and what they’re looking for:

• AI detectors try to find text that looks like it was generated by an AI writing tool. They do this by measuring specific characteristics of the text (perplexity and burstiness) – not by comparing it to a database.
• Plagiarism checkers try to find text that is copied from a different source. They do this by comparing the text to a large database of previously published sources, student theses, and so on, and detecting similarities – not by measuring specific characteristics of the text.

However, we’ve found that plagiarism checkers do flag parts of AI-generated texts as plagiarism. This is because AI writing draws on sources that it doesn’t cite. While it usually generates original sentences, it may also include sentences directly copied from existing texts, or at least very similar.

This is most likely to happen with popular or general-knowledge topics and less likely with more specialised topics that have been written about less. Moreover, as more AI-generated text appears online, AI writing may become more likely to be flagged as plagiarism – simply because other similarly worded AI-generated texts already exist on the same topic.

So, while plagiarism checkers aren’t designed to double as AI detectors, they may still flag AI writing as partially plagiarised in many cases. But they’re certainly less effective at finding AI writing than an AI detector.

What are AI detectors used for?

AI detectors are intended for anyone who wants to check whether a piece of text might have been generated by AI. Potential users include:

• Educators (teachers and university instructors) who want to check that their students’ work is original
• Publishers who want to ensure that they only publish human-written content
• Recruiters who want to ensure that candidates’ cover letters are their own writing
• Web content writers who want to publish AI-generated content but are concerned that it may rank lower in search engines if it is identified as AI writing
• Social media moderators, and others fighting automated misinformation, who want to identify AI-generated spam and fake news

Because of concerns about their reliability, most users are reluctant to fully rely on AI detectors for now, but they are already gaining popularity as an indication that a text was AI-generated when the user already had their suspicions.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Detecting AI writing manually

As well as using AI detectors, you can also learn to spot the identifying features of AI writing yourself. It’s difficult to do so reliably – human writing can sometimes seem robotic, and AI writing tools are more and more convincingly human – but you can develop a good instinct for it.

The specific criteria that AI detectors use – low perplexity and burstiness – are quite technical, but you can try to spot them manually by looking for text:

• That reads monotonously, with little variation in sentence structure or length
• With predictable, generic word choices and few surprises

You can also use approaches that AI detectors don’t, by watching out for:

• Overly polite language: Chatbots like ChatGPT are designed to play the role of a helpful assistant, so their language is very polite and formal by default – not very conversational.
• Hedging language: Look for a lack of bold, original statements and for a tendency to overuse generic hedging phrases: “It’s important to note that …” “X is widely regarded as …” “X is considered …” “Some might say that …”
• Inconsistency in voice: If you know the usual writing style and voice of the person whose writing you’re checking (e.g., a student), then you can usually see when they submit something that reads very differently from how they normally write.
• Unsourced or incorrectly cited claims: In the context of academic writing, it’s important to cite your sources. AI writing tools tend not to do this or to do it incorrectly (e.g., citing nonexistent or irrelevant sources).
• Logical errors: AI writing, although it’s increasingly fluent, may not always be coherent in terms of its actual content. Look for points where the text contradicts itself, makes an implausible statement, or presents disjointed arguments.

In general, just trying out some AI writing tools, seeing what kinds of texts they can generate, and getting used to their style of writing are good ways to improve your ability to spot text that may be AI-generated.

AI image and video detectors

AI image and video generators such as DALL-E, Midjourney, and Synthesia are also gaining popularity, and it’s increasingly important to be able to detect AI images and videos (also called “deepfakes”) to prevent them from being used to spread misinformation.

Due to the technology’s current limitations, there are some obvious giveaways in a lot of AI-generated images and videos: anatomical errors like hands with too many fingers; unnatural movements; inclusion of nonsensical text; and unconvincing faces.

But as these AI images and videos become more advanced, they may become harder to detect manually. Some AI image and video detectors are already out there: for example, Deepware, Intel’s FakeCatcher, and Illuminarty. We haven’t tested the reliability of these tools.

Frequently asked questions

How accurate are AI detectors?

AI detectors aim to identify the presence of AI-generated text (e.g., from ChatGPT) in a piece of writing, but they can’t do so with complete accuracy. In our comparison of the best AI detectors, we found that the 10 tools we tested had an average accuracy of 60%. The best free tool had 68% accuracy, the best premium tool 84%.

Because of how AI detectors work, they can never guarantee 100% accuracy, and there is always at least a small risk of false positives (human text being marked as AI-generated). Therefore, these tools should not be relied upon to provide absolute proof that a text is or isn’t AI-generated. Rather, they can provide a good indication in combination with other evidence.

How can I detect AI writing?

Tools called AI detectors are designed to label text as AI-generated or human. AI detectors work by looking for specific characteristics in the text, such as a low level of randomness in word choice and sentence length. These characteristics are typical of AI writing, allowing the detector to make a good guess at when text is AI-generated.

But these tools can’t guarantee 100% accuracy. Check out our comparison of the best AI detectors to learn more.

You can also manually watch for clues that a text is AI-generated – for example, a very different style from the writer’s usual voice or a generic, overly polite tone.

Can I use AI tools to write my essay?

Using AI writing tools (like ChatGPT) to write your essay is usually considered plagiarism and may result in penalisation, unless it is allowed by your university. Text generated by AI tools is based on existing texts and therefore cannot provide unique insights. Furthermore, these outputs sometimes contain factual inaccuracies or grammar mistakes.

However, AI writing tools can be used effectively as a source of feedback and inspiration for your writing (e.g., to generate research questions). Other AI tools, like grammar checkers, can help identify and eliminate grammar and punctuation mistakes to enhance your writing.

When was ChatGPT released?

ChatGPT was publicly released on 30 November 2022. At the time of its release, it was described as a “research preview”, but it is still available now, and no plans have been announced so far to take it offline or charge for access.

ChatGPT continues to receive updates adding more features and fixing bugs. The most recent update at the time of writing was on 24 May 2023.

How long will ChatGPT be free?

It’s not clear whether ChatGPT will stop being available for free in the future – and if so, when. The tool was originally released in November 2022 as a “research preview”. It was released for free so that the model could be tested on a very large user base.

The framing of the tool as a “preview” suggests that it may not be available for free in the long run, but so far, no plans have been announced to end free access to the tool.

A premium version, ChatGPT Plus, is available for £16 a month and provides access to features like GPT-4, a more advanced version of the model. It may be that this is the only way OpenAI (the publisher of ChatGPT) plans to monetise it and that the basic version will remain free. Or it may be that the high costs of running the tool’s servers lead them to end the free version in the future. We don’t know yet.