Researchers from the University of Kansas have developed a tool capable of identifying artificial intelligence (AI)-generated academic science writing with over 99% accuracy.

With the advent of AI-generated text from sources like OpenAI’s ChatGPT — a text-generating chatbot — University of Kansas researchers determined that certain signs signal when academic writing has been created using a chatbot. As such, the researchers took those signs and used them to develop a detection tool for identifying when those papers are written by a chatbot.

Source: University of Kansas

To build this tool, the researchers analyzed a type of article called perspectives that offers an overview of specific research topics written by scientists. After selecting 64 perspectives, the researchers created 128 ChatGPT-generated articles on those same research topics for the purpose of training the model. Upon a comparison of the articles, it was discovered that one indicator of the AI-generated content was the predictability of the writing, according to the researchers.

While human writing tends to feature more complex paragraph structures that vary in the number of sentences, total words per paragraph and fluctuating sentence length, AI-generated writing reportedly does not. Further, punctuation marks and vocabulary favored by chatbots are also reportedly giveaways of AI-generated text, with the researchers identifying 20 such characteristics that the models have been trained to look for.

During tests of the tool, the researchers found that the model accurately identified AI-generated full perspective articles with a 100% rate of accuracy and individual paragraphs within the article with a 92% rate of accuracy. Additionally, the model reportedly outperformed an online AI text detector by a significant margin.

An article detailing the team’s findings, “Distinguishing academic science writing from humans or ChatGPT with over 99% accuracy using off-the-shelf machine learning tools,” appears in the journal Cell Reports Physical Science.