By analyzing neural signals, a brain-computer interface (BCI) can now almost instantaneously synthesize the speech of a man who lost use of his voice due to a neurodegenerative disease, a new study finds.
The researchers caution it will still be a long time before such a device, which could restore speech to paralyzed patients, will find use in everyday communication. Still, the hope is this work “will lead to a pathway for improving these systems further—for example, through technology transfer to industry,” says Maitreyee Wairagkar, a project scientist at the University of California Davis’s Neuroprosthetics Lab.
A major potential application for brain-computer interfaces is restoring the ability to communicate to people who can no longer speak due to disease or injury. For instance, scientists have developed a number of BCIs that can help translate neural signals into text.
However, text alone fails to capture many key aspects of human speech, such as intonation, that help to convey meaning. In addition, text-based communication is slow, Wairagkar says.
Now, researchers have developed what they call a brain-to-voice neuroprosthesis that can decode neural activity into sounds in real time. They detailed their findings 11 June in the journal Nature.
“Losing the ability to speak due to neurological disease is devastating,” Wairagkar says. “Developing a technology that can bypass the damaged pathways of the nervous system to restore speech can have a big impact on the lives of people with speech loss.”
Neural Mapping for Speech Restoration
The new BCI mapped neural activity using four microelectrode arrays. In total, the scientists placed 256 microelectrode arrays in three brain regions, chief among them the ventral precentral gyrus, which plays a key role in controlling the muscles underlying speech.
“This technology does not ‘read minds’ or ‘read inner thoughts,’” Wairagkar says. “We record from the area of the brain that controls the speech muscles. Hence, the system only produces voice when the participant voluntarily tries to speak.”
The researchers implanted the BCI in a 45-year-old volunteer with amyotrophic lateral sclerosis (ALS), the neurodegenerative disorder also known as Lou Gehrig’s disease. Although the volunteer could still generate vocal sounds, he was unable to produce intelligible speech on his own for years before the BCI.
The neuroprosthesis recorded the neural activity that resulted when the patient attempted to read sentences on a screen out loud. The scientists then trained a deep-learning AI model on this data to produce his intended speech.
The researchers also trained a voice-cloning AI model on recordings made of the patient before his condition so the BCI could synthesize his pre-ALS voice. The patient reported that listening to the synthesized voice “made me feel happy, and it felt like my real voice,” the study notes.
Neuroprosthesis Reproduces a Man’s Speech UC Davis
In experiments, the scientists found that the BCI could detect key aspects of intended vocal intonation. They had the patient attempt to speak sets of sentences as either statements, which had no changes in pitch, or as questions, which involved rising pitches at the ends of the sentences. They also had the patient emphasize one of the seven words in the sentence “I never said she stole my money” by changing its pitch. (The sentence has seven different meanings, depending on which word is emphasized.) These tests revealed increased neural activity toward the ends of the questions and before emphasized words. In turn, this let the patient control his BCI voice enough to ask a question, emphasize specific words in a sentence, or sing three-pitch melodies.
“Not only what we say but also how we say it is equally important,” Wairagkar says. “Intonation of our speech helps us to communicate effectively.”
All in all, the new BCI could acquire neural signals and produce sounds with a delay of 25 milliseconds, enabling near-instantaneous speech synthesis, Wairagkar says. The BCI also proved flexible enough to speak made-up pseudo-words, as well as interjections such as “ahh,” “eww,” “ohh,” and “hmm.”
The resulting voice was often intelligible, but not consistently so. In tests where human listeners had to transcribe the BCI’s words, they understood what the patient said about 56 percent of the time, up from about 3 percent from when he did not use the BCI.
Neural recordings of the BCI participant shown on screen.UC Davis
“We do not claim that this system is ready to be used to speak and have conversations by someone who has lost the ability to speak,” Wairagkar says. “Rather, we have shown a proof of concept of what is possible with the current BCI technology.”
In the future, the scientists plan to improve the accuracy of the device—for instance, with more electrodes and better AI models. They also hope that BCI companies might start clinical trials incorporating this technology. “It is yet unknown whether this BCI will work with people who are fully locked in”—that is, nearly completely paralyzed, save for eye motions and blinking, Wairagkar adds.
Another interesting research direction is to study whether such speech BCIs could be useful for people with language disorders, such as aphasia. “Our current target patient population cannot speak due to muscle paralysis,” Wairagkar says. “However, their ability to produce language and cognition remains intact.” In contrast, she notes, future work might investigate restoring speech to people with damage to brain areas that produce speech, or with disabilities that have prevented them from learning to speak since childhood.
