User state estimation systems and methods
Inventors
Durkee, Kevin • Pappada, Scott • Ortiz, Andres • DePriest, William • Feeney, John • Geyer, Alexandra • Sullivan, Seamus • Wiggins, Sterling
Assignees
Publication Number
US-11864896-B2
Publication Date
2024-01-09
Expiration Date
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
Computer based systems and methods for estimating a user state are disclosed. In some embodiments, the methods comprise inputting a first input at an intermittent interval and a second input at a frequent interval into a user state estimation model to estimate the user state. In some embodiments, the first inputs are enhanced by injecting a noise input to create a plurality of enhanced first inputs whereby the plurality of enhance first inputs correspond to the plurality of second inputs at the frequent interval. In some embodiments, the first input comprises a self-reported input and the second inputs comprise a physiological input, a performance input or a situational input. In some embodiments, a machine learning algorithm creates the state estimation model. In some embodiments, the state estimation model estimates a future user state. In some embodiments, a computer based system for estimating a user state is provided.
Core Innovation
Computer based systems and methods for estimating a user state are disclosed that input a first input at an intermittent interval and a second input at a frequent interval into a user state estimation model to estimate the user state. In some embodiments, the first inputs are enhanced by injecting a noise input to create a plurality of enhanced first inputs whereby the plurality of enhanced first inputs correspond to the plurality of second inputs at the frequent interval. In some embodiments, the first input comprises a self-reported input and the second inputs comprise a physiological input, a performance input or a situational input. In some embodiments, a machine learning algorithm creates the state estimation model and the state estimation model estimates a future user state.
The background identifies limitations of prior user state estimation approaches including low resolution, infrequent updates, and poor generalizability across human operators. The disclosed methods address these limitations by injecting physiological-derived noise into self-reported user state ratings, fusing multi-modal data sources, retaining memory of each data source, and applying on-line adaptation and prediction to produce real-time, high resolution (0-100) user state estimates. The methods utilize machine learning approaches such as ANNs and SVMs and may collect baseline physiological data and perform on-line retraining to adapt weights based on incoming operator data. The disclosed framework is intended to provide continuous, frequent estimates and predictive capabilities to support adaptive aiding and interface modification.
Claims Coverage
Independent claims identified: 1, 14, 27, and 29. Four inventive features are extracted corresponding to these independent claims.
Estimating user functional state using intermittent and frequent inputs
receiving a plurality of first inputs at an intermittent interval; receiving a plurality of second inputs at a frequent interval; inputting the plurality of first inputs and the plurality of second inputs into a functional state estimation model whereby a user functional state can be estimated based on the plurality of second inputs; wherein the user functional state comprises an objective measure cognitive state of a trainee in a computer lased training simulator; wherein the objective measure cognitive state comprises one selected from a group comprising: a real-time engagement of the trainee in a training situation, and a real-time workload of the trainee in the training situation; receiving a plurality of enhanced actual first inputs and a plurality of actual second inputs; retraining the functional state estimation model with the plurality of enhanced actual first inputs and the plurality of actual second inputs to create a retrained functional state estimation model; and estimating the user functional state with the retrained functional state estimation model.
Presenting and adapting a training interface based on estimated cognitive state
receiving a plurality of first inputs at an intermittent interval; receiving a plurality of second inputs at a frequent interval; inputting the plurality of first inputs and the plurality of second inputs into a functional state estimation model whereby a user functional state can be estimated based on the plurality of second inputs; receiving a plurality of enhanced actual first inputs and a plurality of actual second inputs; retraining the functional state estimation model with the plurality of enhanced actual first inputs and the plurality of actual second inputs to create a retrained functional state estimation model; estimating the user functional state with the retrained functional state estimation model; presenting a training situation to the trainee through a user interface, and adapting an interface based on the objective measure of the cognitive state of the trainee.
Computer based system implementing functional state estimation with retraining
a computer based system comprising a processor and a non-transitory computer readable medium having a computer readable program code embodied therein, said computer readable program code configured to be executed to implement a method comprising: receiving a plurality of first inputs at an intermittent interval; receiving a plurality of second inputs at a frequent interval; inputting the plurality of first inputs and the plurality of second inputs into a functional state estimation model whereby a user functional state can be estimated based on the plurality of second inputs; wherein the user functional state comprises an objective measure of a cognitive state of a trainee in a computer based training simulator; receiving a plurality of enhanced actual first inputs and a plurality of actual second inputs; retraining the functional state estimation model with the plurality of enhanced actual first inputs and the plurality of actual second inputs to create a retrained functional state estimation model; and estimating the user functional state with the retrained functional state estimation model.
System presenting and adapting training interface based on estimated state
a computer based system comprising a processor and a non-transitory computer readable medium having a computer readable program code embodied therein, said computer readable program code configured to be executed to implement a method comprising: receiving a plurality of first inputs at an intermittent interval; receiving a plurality of second inputs at a frequent interval; inputting the plurality of first inputs and the plurality of second inputs into a functional state estimation model whereby a user functional state can be estimated based on the plurality of second inputs; wherein the estimating of the user functional state comprises an objective measure of a cognitive state of a trainee in a computer based training simulator and the method further comprises: presenting a training situation to the trainee through a user interface; and adapting an interface based on the objective measure of the cognitive state of the trainee; receiving a plurality of enhanced actual first inputs and a plurality of actual second inputs; retraining the functional state estimation model with the plurality of enhanced actual first inputs and the plurality of actual second inputs to create a retrained functional state estimation model; and estimating the user functional state with the retrained functional state estimation model.
The independent claims focus on (1) receiving intermittent self-reported inputs and frequent physiological/performance/situational inputs, (2) injecting or receiving enhanced first inputs and retraining the functional/state estimation model with enhanced and actual inputs, (3) estimating current and future user functional/cognitive states (including real-time engagement and workload for a trainee in a training simulator), and (4) implementing these methods and adaptations in computer based systems and user interfaces.
Stated Advantages
Provides a real-time, model-based classifier and predictor of user state on a continuous, high resolution (0-100) scale, in some embodiments second-by-second.
Enables more frequent estimates of user state (as frequently as once per second) while maintaining acceptable accuracy by using multi-modal data sources and retaining memory of each data source.
Improves generalization across users and allows on-line adaptation without requiring custom-built classifiers for each human operator through on-line learning and real-time weight adjustments.
Supports predictive estimation of future user state and its relationship to performance to facilitate proactive adaptive aiding and interface modification.
Allows injection of physiological-derived noise into intermittent self-reported inputs to generate higher resolution training inputs that correspond to frequent interval objective inputs.
Documented Applications
Estimating user states such as workload, task engagement, stress or trust.
Assessing an objective measure cognitive state of a trainee in a computer based training simulator, including real-time engagement and real-time workload.
Application in the remotely piloted aircraft (RPA) task domain.
Adaptive aiding and adaptive automation including modifying a user interface based on current or predicted user state to augment user performance.
Use in scenarios such as Suppression of Enemy Defenses (SEAD) missions to track information processing stages and inform adaptive aids.
Integration with learning management systems as illustrated in the patent (see FIG. 9).
Interested in licensing this patent?