An emerging artificial intelligence (AI) system called a "semantic decoder" can turn a person's brain activity into a continuous stream of text while they listen to a story or think about sharing a story. Researchers at The University of Texas at Austin made a system that could help people who are mentally aware but physically unable to speak, like those who have had strokes and lost their speech, to communicate more effectively.

The study, published in Nature Neuroscience, relies in part on a transformer model, similar to the ones that power Open AI’s ChatGPT and Google’s Bard.

Unlike other systems developed on to decode language, this one doesn't require surgical implants. Participants are not required to only use words from a list, either. An fMRI machine measures brain activity after the decoder has been trained for a long time by the person listening to hours of podcasts while in the scanner. Later, if the person is willing to have their thoughts decoded, they can listen to a new story or imagine telling a story, and the machine will be able to figure out what the story is just from their brain activity.

[Learn more about artificial intelligence at GlobalSpec]

“For a noninvasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences,” said researcher Alexander Huth. “We’re getting the model to decode continuous language for extended periods of time with complicated ideas.”

A work in progress

The result is not an exact copy of what was said. Instead, it was made to get the main idea of what is being said or thought, even if it isn't perfect. About half of the time, when the decoder has been taught to watch the brain activity of a participant, the tool produces content that closely (and sometimes exactly) matches the original words' meanings.

The method cannot be used outside of a lab at present because it needs time on an fMRI machine. But experts think that this work could be used in other brain imaging systems that are easier to carry around, such as functional near-infrared spectroscopy (fNIRS).

To contact the author of this article, email ccooney@globalspec.com