Systems and methods for decoding intended speech from neuronal activity
Inventors
Stavisky, Sergey • Shenoy, Krishna V. • Henderson, Jaimie M.
Assignees
Leland Stanford Junior University • Vitesco Technologies USA LLC
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12431136-B2
Publication Date
2025-09-30
Expiration Date
Abstract
Systems and methods for decoding indented speech from neuronal activity in accordance with embodiments of the invention are illustrated. One embodiment includes a neuronal speech system for decoding intended speech from neuronal signals includes a neuronal signal recorder implanted into a user's brain, including a multielectrode array, controller circuitry, and a communication circuitry capable of transmitting data to a neuronal signal decoder, the neuronal signal decoder located externally from the user's body, including a processor, an input/output interface, and a memory, where the memory contains a neuronal speech application that directs the processor to obtain neuronal signal data from the neuronal signal recorder, where the neuronal signal data describes neuronal activity proximal to the implanted neuronal signal recorder, decode speech features from the neuronal signal data using a neuronal decoding model, construct a speech string from the decoded speech features; and output the constructed speech string via an output device.
Core Innovation
An external neuronal speech decoder system receives neuronal signals from an implanted multielectrode array recorder in a user’s brain. The system extracts neuronal features, including action-potential/spike features via voltage-threshold filtering and local field potential features via band-pass filtering and band power/time-window aggregation, and uses the extracted neural features to decode speech features with a neuronal decoding model together with a language model.
The language model adjusts prior probabilities of different speech features based on previous speech feature probabilities produced by the neuronal decoding model. The system constructs an operational speech string from the decoded operational speech features and provides the speech string to a user via a vocalizer and/or a display device and text-based messaging system. The system determines whether the language model has degraded based on adherence to a statistical language model.
In response to a determination of degradation, the system initializes a closed-loop training phase using a second prompt to attempt to speak a given phrase. Closed-loop neuronal signal data is obtained during the second prompt, speech features are decoded using the neuronal decoding model and language model, and a speech string is constructed; when the constructed speech string does not match the second prompt, the language model is retrained using the closed-loop neuronal signal data and the second prompt.
The description supports training that includes an open-loop training phase followed by operational decoding outside of a training phase. The system further supports adapting models based on implant location and an output target, and it tailors processing based on implant location such as motor cortex hand/arm region, Broca’s area, and Wernicke’s area. The description also contrasts with conventional large ECoG electrode grids and supports wireless telemetry through on-implant preprocessing.
Claims Coverage
The patent includes two independent claims, one for a brain-machine interface for speech and one for a method for speech decoding. Across these independent claims, the core coverage centers on open-loop training with a prompt to obtain neuronal data, decoding with a neuronal decoding model and a language model that adjusts speech-feature priors, operational decoding and speech-string construction, and degradation detection with adherence to a statistical language model followed by closed-loop training and language-model retraining when mismatch occurs.
Open-loop training with prompt and neuronal data collection
Initialize an open-loop training phase by providing a prompt to the user to attempt to speak a given phrase and obtaining open-loop neuronal signal data from an array of microelectrodes configured to be implanted into a human user's brain, where the open-loop neuronal signal data is recorded during the providing of the prompt.
Neuronal decoding model with speech-feature probabilities and language-model prior adjustment
Train a neuronal decoding model and a language model to decode speech features from neuronal signal data using the provided prompt and the open-loop neuronal signal data, where the neuronal decoding model generates speech feature probabilities from neuronal signal data and the language model adjusts prior probabilities of different speech features based on previous speech feature probabilities produced by the neuronal decoding model.
Operational decoding outside training phase and speech-string construction
Obtain operational neuronal signal data from the array of microelectrodes, where the operational neuronal signal data is recorded outside of a training phase, decode operational speech features from the operational neuronal signal data using the neuronal decoding model and the language model, and construct an operational speech string from the decoded operational speech features.
Language-model degradation determination and closed-loop retraining via prompt mismatch
Determine if the language model has degraded based on adherence to a statistical language model; in response to a determination of degradation, initialize a closed-loop training phase by providing a second prompt to the user and obtaining closed-loop neuronal signal data recorded during the providing of the prompt, decoding speech features from the closed-loop neuronal signal data using the neuronal decoding model and the language model, constructing a speech string from the decoded speech features, providing the speech string to the user, and when the speech string does not match the second prompt, retrain the language model using the closed-loop neuronal signal data and the second prompt.
Both independent claims focus on training and operational decoding using a neuronal decoding model that produces speech feature probabilities and a language model that adjusts prior probabilities, together with degradation determination based on adherence to a statistical language model and closed-loop retraining of the language model based on a mismatch between the constructed speech string and a second prompt.
Stated Advantages
Provides operational speech decoding from neuronal signals by constructing an operational speech string from decoded speech features.
Adapts the language model during closed-loop training when degradation is detected and when the constructed speech string does not match the second prompt.
Determines language-model degradation based on adherence to a statistical language model.
Documented Applications
Neuronal speech decoding using a brain-machine interface to decode speech features and construct speech strings for output to a user.
Interested in licensing this patent?