Human-computer interface using high-speed and accurate tracking of user interactions
Inventors
ALCAIDE, Ramses • Padden, Dereck • JANTZ, Jay • HAMET, James • MORRIS, Jr., Jeffrey • Pereira, Arnaldo
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-10664050-B2
Publication Date
2020-05-26
Expiration Date
Abstract
Embodiments described herein relate to systems, devices, and methods for use in the implementation of a human-computer interface using high-speed, and efficient tracking of user interactions with a User Interface/User Experience that is strategically presented to the user. Embodiments described herein also relate to the implementation of a hardware agnostic human-computer interface that uses neural, oculomotor, and/or electromyography signals to mediate user manipulation of machines and devices.
Core Innovation
The disclosed system determines a point of focus of a user and an action intended by the user in an interactive environment using eye-movement signals recorded by an eye-tracker that includes at least two sensors. The system receives eye-movement signals from each sensor and converts the sensor-specific signals into gaze vectors associated with each sensor, where each gaze vector indicates a gaze angle of the eye of the user.
The core innovation calibrates independently recorded eye-movement signals by determining a degree of obliqueness of each gaze vector relative to a vertical angle associated with that sensor. Based on the degree of obliqueness, the processor determines a weight associated with each sensor, applies the set of weights to the plurality of sets of eye-movement signals to determine calibrated eye-movement signals, and uses the calibrated eye-movement signals to determine the point of focus of the user.
The disclosed approach further includes resolving gaze vectors along axes and computing average gaze vectors to form a calibrated gaze vector, and replacing missing data points using a kinematics model of simulated eye-movements. In addition, a neural recording device receives EEG signals, determines an expected point of focus, computes an error by comparing the calculated point of focus and the expected point of focus, and corrects the calculated point of focus to produce a calibrated point of focus.
Claims Coverage
The document includes four independent claims, each centered on sensor-specific gaze-vector calibration using degrees of obliqueness and weights, followed by determining a point of focus and an intended action. Across the independent claims, the inventive features include multi-sensor eye-movement recording, gaze-vector computation, obliqueness-based weighting for calibrated eye-movement signals, and mapping focus to an implemented action; refinements include neural-signal calibration, missing-data replacement via a kinematics model, and axis-resolved calibrated gaze vectors.
Multi-sensor eye-movement recording with calibrated gaze vectors via obliqueness weighting
A display presents an interactive environment; an eye-tracker coupled to the display records plurality of sets of eye-movement signals from an eye using at least two sensors; the processor receives the eye-movement signals and computes, for each sensor, a gaze vector indicating a gaze angle; it determines a degree of obliqueness for each gaze vector relative to a vertical angle associated with that sensor; it determines a weight associated with each sensor based on the degree of obliqueness; it applies the set of weights to determine calibrated eye-movement signals; it determines a point of focus based on the calibrated eye-movement signals; it determines an action intended by the user based on the point of focus; and it implements the action intended by the user.
Empirically predetermined obliqueness-weighted calibration for point-of-focus and action
An apparatus includes a display presenting an interactive environment and an eye-tracker coupled to the display with at least two sensors recording independently a plurality of sets of eye-movement signals; the processor receives the eye-movement signals, generates and presents a stimulus via the interactive environment and the display, computes gaze vectors associated with each sensor indicating a gaze angle, determines a degree of obliqueness relative to a vertical angle for each gaze vector, determines a weight associated with each sensor based on the degree of obliqueness and an empirically pre-determined weighting function, applies the set of weights to determine calibrated eye-movement signals, determines a point of focus based on the calibrated eye-movement signals, determines an action intended by the user based on the point of focus, and implements the action intended by the user.
Missing data replacement using a kinematics model to produce calibrated eye-movement signals for action
An apparatus includes a display presenting an interactive environment and an eye-tracker coupled to the display with at least two sensors recording independently plurality of sets of eye-movement signals; the processor receives the eye-movement signals, generates and presents a stimulus via the interactive environment and the display, identifies a set of missing data points in the plurality of sets of eye-movement signals, receives information related to the at least two sensors, generates based on the information a kinematics model of a set of simulated eye-movements of a simulated user, computes simulated eye-movement signals associated with each sensor using the kinematics model, computes a set of replacement data points to replace the missing data points based on the simulated eye-movement signals, incorporates the replacement data points to generate calibrated eye-movement signals associated with each sensor, determines a point of focus based on the calibrated eye-movement signals, determines an action intended by the user based on the point of focus, and implements the action intended by the user.
Stimulus-driven interactive interface action from calibrated eye-movement signals using obliqueness weights
A method presents, to a user and via a display, a stimulus in an interactive user interface; it receives eye-movement signals from an eye-tracker recorded independently by at least two sensors positioned on the eye-tracker; it receives information related to the presented stimulus; it computes based on the eye-movement signals a gaze vector associated with each sensor indicating a gaze angle; it determines a degree of obliqueness of each gaze vector relative to a vertical angle associated with that sensor; it defines weights associated with each sensor based on the degree of obliqueness to generate a set of weights; it applies the set of weights to the eye-movement signals to compute calibrated eye-movement signals for determining a point of focus; it determines the point of focus, determines an action intended by the user based on the point of focus and the stimulus, and implements the action via the interactive user interface.
Axis-resolved calibrated gaze vector from grouped sensor gaze vectors for point-of-focus and action
A method presents to a user and via a display a stimulus in an interactive user interface; it receives eye-movement signals from an eye-tracker recorded independently by at least two sensors; it computes a set of gaze vectors with each gaze vector associated with each sensor from at least four sensors; it resolves the set of gaze vectors along a first axis and a second axis orthogonal to the first axis; it computes a first average gaze vector based on a first weighted average of gaze vectors associated with a first set of sensors grouped along the first axis; it computes a second average gaze vector based on a second weighted average of gaze vectors associated with a second set of sensors grouped along the second axis; it computes a calibrated gaze vector based on the first average gaze vector and the second average gaze vector; it determines a point of focus based on the calibrated gaze vector; it determines an action intended by the user based on the point of focus and the stimulus; and it implements the action via the interactive user interface.
Across the independent claims, the document provides multi-sensor eye-tracker processing that computes sensor-associated gaze vectors, determines degrees of obliqueness relative to sensor-specific vertical angles, derives sensor weights, and applies the weights to produce calibrated eye-movement signals or a calibrated gaze vector for determining a point of focus. The point of focus is then used with the stimulus to determine an action intended by the user and implement the action in the interactive user interface. Additional independent-claim coverage includes replacement of missing data points using a kinematics model of simulated eye-movements and calibration using axis-resolved grouped sensor gaze vectors.
Stated Advantages
Documented Applications
Use of the determined point of focus and the stimulus to determine and implement an action intended by the user via an interactive user interface.
Use of calibrated gaze determinations as part of gaze-based interaction within an interactive environment presented on a display.
Interested in licensing this patent?