Brain-computer interface with high-speed eye tracking features
Inventors
ALCAIDE, Ramses • Padden, Dereck • JANTZ, Jay • HAMET, James • MORRIS, Jr., Jeffrey • Pereira, Arnaldo
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11269414-B2
Publication Date
2022-03-08
Expiration Date
Abstract
Embodiments described herein relate to systems, devices, and methods for use in the implementation of a brain-computer interface that integrates real-time eye-movement tracking with brain activity tracking to present and update a user interface that is strategically designed for high speed and accuracy of human-machine interaction. Embodiments described herein also relate to the implementation of a hardware agnostic brain-computer interface that uses real-time eye tracking and online analysis of neural signals to mediate user manipulation of machines.
Core Innovation
The invention describes a hybrid brain-computer interface apparatus that is hardware-agnostic and provides high-speed, accurate human-machine interaction by combining real-time video-based eye tracking and neural recording. A display presents an interactive environment to a user, and an eye-tracker records eye-movement signals while a neural recording device records neural signals generated by the user. An interfacing device receives both modalities, generates and presents a stimulus, and uses eye-movement and neural signals to drive control of an interactive user interface.
The approach determines a pointing control signal based on the eye-movement signals and an action control signal based on the neural signals. A composite set of data is generated by combining a first portion based on the pointing control signal and a second portion based on the action control signal. The composite set of data is provided to a machine learning model to determine a point of focus of the user, and an action intended by the user is identified based on the point of focus of the user.
The system supports neural signatures such as ERPs including P300 and visually evoked potentials including VEP/SSVEP/TVEP, as well as other neural signal options such as motor imagery signals and brain state dependent signals. For eye-movement processing, it uses high-speed, low-latency gaze classification with lag-less filtering and machine-learning, including gaze classification with gaze-kinematics models such as dual Kalman estimation. A training and prediction pipeline is used to handle missing or low sampling gaze data by estimating missing gaze data points and incorporating replacement points, enabling continued real-time control.
Claims Coverage
The partial content provides three independent claims (apparatus, non-transitory processor-readable medium, and method) that cover a hybrid eye-tracker plus neural-recording interactive environment workflow. Four inventive features are emphasized across the independent claims.
Hybrid interface for pointing and action control
A display presenting an interactive environment to a user, an eye-tracker recording eye-movement signals, a neural recording device recording neural signals, and an interfacing device that determines a pointing control signal from the eye-movement signals and an action control signal from the neural signals.
Composite data fusion for point of focus
A processor generates a composite set of data by combining a first portion based on the pointing control signal and a second portion based on the action control signal and provides the composite set of data to a machine learning model.
Machine learning point of focus determination and intended action identification
The processor uses the machine learning model to determine a point of focus of the user based on the composite set of data, and identifies an action intended by the user based on the point of focus of the user.
Stimulus-driven interactive environment action implementation
The system presents a stimulus via the interactive environment to the user, and implements the determined action intended by the user via the interactive user environment.
Across the independent claims, the core coverage is a hardware-agnostic hybrid human-machine interface that fuses an eye-derived pointing control portion with a neural-derived action control portion into a composite set of data for a machine learning model that determines a point of focus. Based on the point of focus and stimulus context, the system determines an intended action and implements it in the interactive environment.
Stated Advantages
high-speed, accurate human-machine interaction
high-speed, low-latency gaze classification for real-time control
improved real-time control via handling missing or low sampling gaze data
Documented Applications
Driving a strategically designed interactive user interface based on a user’s point of focus and intended action.
Use of stimulus presentation matrices including speller-style selection and interaction modes (hold-release and indirect selection) in the interactive environment.
Interested in licensing this patent?