AI Content Detection: How ChatGPT and AI-Generated Text Is Found

The following article discusses how ChatGPT and AI-generated text can be found and recognized by AI Detectors.

AI and human written content can be difficult to distinguish due to the current advanced AI models. For the inexperienced or unaware reader, AI-created content can look like a legit piece of writing. So how is one supposed to detect if a given text is written by AI or a human and is it possible to detect. If so how does these AI-content detectors work?

How Does AI Content Detection Work?

There are many tools and detection programs, which can detect if a text is written by AI or a human-being.

These tools work in several different ways to determine if a text is generated by AI or not. The overarching categories for detection methods are:

• Linguistic analysis: A text generated by AI may have characteristics such as lack of semantic meaning or a tendency to repeat itself.

• Comparison with known AI-generated text: If a text resembles a previously known AI-generated text, it may be a sign that this text is also generated by AI.

Within these two categories, there are multiple different techniques. Some are a mix of the two.

‍

The 4. techniques for identifying a text as AI generated

AI-generated text is increasingly becoming more sophisticated and cannot always be detected or distinguished from human-written text.

"There is no magical solution to AI detection. Just like humans, as these models become more powerful, these detection models are playing catch-up and they’re not going to be as good"
Irene Solaiman, policy director at AI start-up Hugging Face.
Source: EuroNews

The AI detectors use a combination of natural language processing (NLP) techniques and machine learning algorithms to identify patterns and features that are common in AI-generated text.

Here are a few examples of how an AI detection tool works:

1. Classifiers: A classifier can be trained to recognize text generated by a specific model by analyzing the language patterns of the text.

2. Embeddings: Embeddings is a way of representing data so similar data point can be clustered together. Embeddings is used as a model to identify the text.

3. Perplexity: Perplexity is a measure of the model's "surprise" or "uncertainty" when it encounters a new piece of text. Also called the randomness or the complexity of a given text.

4. Burstiness: Burstiness is comparing the variation of the sentences, like det different words etc. With algorithms to help identifying when a burst occurs.

‍

1. Using a classifier for AI detection

A classifier is a machine learning model used to categorize or classify data into predefined classes.

A classifier uses various features of the text to learn the patterns and characteristics that are typically found in AI-generated text compared to human-written text such as.

A classifier uses features such as:

• The words

• The grammar

• The style

• The tone

Once the language patterns are identified, they can be used as input to a classifier.

A classifier is a type of computer program that can recognize patterns in data. It works like a sorting machine, taking in data and then sorting it into different categories.

For example, a classifier might be used to sort a pile of apples into different types, such as red apples, green apples, and yellow apples. The classifier looks at each apple and then decides which category it belongs in.

The classifier can then be used to identify new text as being generated by the specific model or not.

The two types of classifiers used for AI-generated content detection are supervised and unsupervised classifiers.

Supervised classifiers are trained on labeled data, meaning that the dataset used to train the classifier is composed of examples that have already been labeled as AI-generated or human-written. The algorithm is trained on labeled data, such as pictures of animals with their respective labels like a cat or a dog, spam mail or not spam mail.

Unsupervised classifiers, on the other hand, are trained on unlabeled data, and they must discover the structure of the dataset by themselves.

An unsupervised classifier is an algorithm that is trained on unlabeled data, where each input is unknown. An example of an unsupervised classifier is it finds patterns or relationships in the data without any prior knowledge of what the groups or clusters should be.

‍

2. Using a embedding for detection

In the context of AI and NLP, embeddings are a way of representing words, phrases, or other language elements in a high-dimensional vector space. These vectors capture the meaning of the words and their relationships to one another.

In content detection, embeddings can be used to represent the words in a text. Then these embeddings can be fed into a machine learning model to classify the text into different categories, such as spam or not spam.

ChatGPT is based on GPT-3 (Generative Pre-trained Transformer 3), a large-scale neural network-based language generation model.

In this context embedding can be subcategories into several language patterns, including:

A) Word frequency analysis: The frequency of specific words in the text can be analyzed to identify text generated by a specific model.

B) N-gram analysis: This approach involves analyzing the frequency of specific sequences of words in the text.

C) Syntactic analysis: This approach involves analyzing the grammar and sentence structure of the text.

D) Semantic analysis: This approach involves analyzing the meaning of the text.

A) Word frequency used for detection

Word frequency detection is analyzing the frequency of a specific word in each text.

This includes repetitive or nonsensical phrases that are not commonly found in human-written text. This analysis is simple to implement in a model but may not provide an accurate result for all models.

An example of a text with a high word frequency is:

"AI technologies are often hard to understand. Companies include AI technologies more often than before. AI technology is the future of the world"

"AI technology" is repeated multiple times in the text, and it is the most frequent word in the text. Therefore, an indication for the detector.

B) N-gram analysis used for detection

N-gram analysis is analyzing the frequency of specific sequences of words in the text.

This way of analyzing provides a more accurate result. However, this analysis is more complex to implement into a specific model.

N-gram is like counting words in a row and pairing them also called. With bigrams (N=2) each and every two words are paired together. With trigrams (N=3) three words are paired together.

By using a bigram (N=2) in a sentence: "I like to eat ice cream"

The bigrams would be "I like","like to","to eat","eat ice","ice cream".

By analyzing the N-grams, like bigram in the text above, it is possible to detect patterns that are not commonly found in human-written text, which could indicate that the text is generated by an AI.

C) Syntactic analysis used for detection

Syntactic analysis is analyzing the grammar and sentence structure of the text.

Syntactic analysis, also known as parsing, is a technique used to analyze the grammatical structure of a sentence.

This involves identifying nouns, verbs adjectives in each sentence.

A sentence like: "The dog chased the cat" has one meaning, meanwhile if the same words are put together in a different order: "Chased dog the cat the" the sentence has a different meaning.

Syntactic analysis detects and looks for meaning and structure between the subject and verb. Therefor is syntactic analysis the relation between the words.

In other words, syntactic analysis is an important method to detect patterns and anomalies in the grammatical structure of text. Syntactic analysis is not alone enough to detect AI-generated content but with a combination of word frequency and n-gram etc. a program can detect rather or not it is AI-generated content.

D) Semantic analysis for detection

Semantic analysis is a technique used to analyze the meaning of a text.

It involves identifying the concepts, entities, and relationships expressed in a text, as a result determining the underlying meaning and quality of the text.

With semantic analysis it can be observed if the text lacks coherence or consistency in the meaning of the text. Like syntactic analysis, semantic analysis is not alone enough to detect AI-generated content.

‍

3. Perplexity used for detection

Perplexity is a measure of how well a probability distribution or a language model is able to predict a sample. In the context of AI-generated content detection, perplexity can be used to evaluate the performance of an AI language model and detect if a text is generated by machine or human.

In the case of AI-generated content, the model will have a lower perplexity because it has seen similar patterns in the data it was trained on

In other words, if the text has high complexity, it's more likely to be human-written. The lower the perplexity, the more likely it's AI-generated.

Lets look at an example:

Human-written text: "The world is facing a climate crisis and urgent action is needed to reduce greenhouse gas emissions and prevent further damage to the planet."

AI-generated text: "Climate change is one of the biggest challenges facing humanity today. The need to reduce emissions and mitigate the effects of global warming is more urgent than ever."

The human-written text is more diverse and harder to predict, which leads to a higher complexity.

‍

4. Detect by burstiness

When AI models generate text, they tend to use certain words and phrases more often than what a human would do, due to the fact that they have seen those words and phrases more often in the data they were trained on.

If you find a text that has words and phrases that are used more often in a short period of time, it could be an indication that it was generated by an AI.

For example, when analyzing text generated by AI, it can be observed that the text may have an overuse of certain words or even a lack of variation. This can be a sign that the text was generated by an AI model, as the model is more likely to repeat words or phrases that it has seen frequently in the training data.

‍

Who is interested in detecting whether an AI or a person makes a text?

You might initially think of schools as one of the places where they would like to know if a student has used their own set of skills to answer or not.

And yes, the ban on ChatGPT and moving back from computers to paper and pen have already started. As an example, New York City has recently banned ChatGPT, which will affects 1.800 public schools, serving more than a million students. Read more

But there are many different types of groups that might want to be able to detect ChatGPT-made content;

Several groups of people and organizations could benefit from being able to detect if a language model like ChatGPT generated a text.

• Researchers in natural language processing and computational linguistics may be interested in understanding how AI models generate text and how to improve their performance.

• Businesses may be interested in detecting AI-generated text in order to identify and eliminate fraudulent activities such as spam, fake reviews or fake news.

• Law enforcement agencies may be interested in detecting AI-generated text in order to identify and eliminate criminal activities such as impersonation, identity fraud and cyberbullying.

• Social media platforms may be interested in detecting AI-generated text in order to identify and remove bots and fake accounts that spread misinformation and propaganda.

• Media and journalism organizations may be interested in detecting AI-generated text in order to identify and eliminate false news and propaganda.

• Government organizations may be interested in detecting AI-generated text in order to identify and eliminate disinformation campaigns and propaganda.

• Anyone that is interested in ensuring the authenticity and reliability of the information they consume may also be interested in detecting AI-generated text.

Is Google detecting ChatGPT?

It's a well-known fact that plagiarism and spam content will affect your rankings in Google.

But what about AI-generated content generated by ChatGPT or platforms like our own SEO.ai?

Does Google try to detect these AI texts and hereafter provide a penalty in the rankings?

In short, there is nothing currently indicating that Google is trying to detect AI content by default. And Google has verified they are not as such against AI-generated content.

‍

We have covered extensively that Google is not against AI content (contrary to what a lot of SEOs believe) and shown how it is guidelines have evolved from being ambiguous to be clear that it's only against spammy content.

It is, however, detecting "Spammy automatically-generated content".

We also covered this in our review of how Google potentially could detect ChatGPT content.

And for you that might use AI content, the best way to keep clear of falling into this category is by ensuring that no matter what you create, it's of high quality.

In our experience, this is best achieved by a mix of AI and human collaboration. In that case it's most likely better content - and as a bonus your content will less come off as AI generated content in the detector tools.

We have found that the best results are achieved when artificial intelligence and human collaboration are combined.

This will create content that is of higher quality and will, as a bonus be less likely to be flagged by detector tools as being AI generated.

‍