Researchers from the University of Kansas have developed an artificial intelligence (AI) text detector that can distinguish scientific essays written by humans from computer-generated content produced by chatbots like ChatGPT.

Although AI detectors exist to distinguish between human-written and AI-generated essays, they do not, according to experts, work well on scientific papers. As such, the researchers looked at scientific journals, specifically those focused on the subject of chemistry, to develop their AI text detector.

Graphical abstract. Source: Cell Reports Physical Science (2023). DOI: 10.1016/j.xcrp.2023.101672

The University of Kansas team trained their AI text detector on journals published by the American Chemical Society. The researchers collected about 100 introductory passages from essays written by professionals in the field and subsequently programmed ChatGPT to write similar introductory passages based on these journal abstracts as well as introductory passages based on report titles alone.

When used to scan three different categories of essays — those written by humans, those generated using introductory passages and those produced using report title prompts only — the AI detector identified both the human-generated and title-prompt generated essays 100% of the time. Meanwhile, the text generated from introductory passages was correctly identified by the AI-text generator 98% of the time.

"Academic publishers are grappling with the rapid and pervasive adoption of new artificial intelligence text generators," the researchers explained. "This new detector will allow the scientific community to assess the infiltration of ChatGPT into chemistry journals, identify the consequences of its use, and rapidly introduce mitigation strategies when problems arise."

An article detailing the team’s AI text detector, “Accurately detecting AI text when ChatGPT is told to write like a chemist,” appears in the journal Cell Reports Physical Science.