Systems and methods to react to environmental input
Inventors
Bruni, Sylvain • Chang, Andy • Carlin, Alan • Marc, Yale • Swanson, Leah • Pratt, Stephanie • Mizrahi, Gilbert
Assignees
Publication Number
US-9293054-B2
Publication Date
2016-03-22
Expiration Date
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
Computer implemented systems and methods of communicating a system reaction to environmental input comprising receiving environmental input, determining a hazard state and a user state from the environmental input, determining a system reaction from the hazard state and the user state and communicating the system reaction to a user interface. In some embodiments, the system reaction comprises a system reaction level and in some embodiments the system reaction level corresponds to a stage of automation. In some embodiments, the user interface is a multimodal interface and in some embodiments the user interface is a haptic interface.
Core Innovation
Computer implemented systems and methods of communicating a system reaction to environmental input comprising receiving environmental input, determining a hazard state and a user state from the environmental input, determining a system reaction from the hazard state and the user state and communicating the system reaction to a user interface. In some embodiments, the system reaction comprises a system reaction level and in some embodiments the system reaction level corresponds to a stage of automation. In some embodiments, the user interface is a multimodal interface and in some embodiments the user interface is a haptic interface.
The background identifies a lack of effective flight deck displays that support pilots who will be faced with the challenge of making more complex, strategic decisions in NextGen operations, including integrating different data sources and additional information such as weather into trajectory planning and conflict avoidance. Two key challenges called out are information certainty and multimodal considerations, including uncertainty visualization and the need for technologies that dynamically adjust the mode of communication based upon situational context.
In example embodiments the systems receive environmental input, estimate hazard states (for example using a State Estimation Bayesian Network and a hazard matrix), estimate user states (for example using a predictive index and workload estimator), generate temporal plans or system plans (for example using a Time-Dependent Markov Decision Process (TMDP)), determine automation reactions considering automation and user effectiveness, and communicate system reactions to multimodal user interfaces with system reaction levels that include salience and intrusiveness components to guide alerts and stages of automation.
Claims Coverage
This section identifies three independent claims and extracts the main inventive features from each independent claim (total inventive features: 13).
Receiving multiple environmental inputs
Receiving a first environmental input and a second environmental input; the first environmental input provided by a sensor system on a vehicle.
Hazard state estimation from vehicle sensor input
Determining a hazard state from the first environmental input.
User state determination as user workload
Determining a user state from the second environmental input; the user state comprising a user workload.
Automation reaction determined from effectiveness components
Determining an automation reaction from an automation effectiveness component and a user effectiveness component; the automation reaction comprising at least one of a set of automation responses.
System reaction combining hazard, user and automation inputs
Determining a system reaction from the hazard state, the user state and the automation reaction.
Communicating system reaction to user interface
Communicating the system reaction to a user interface.
Receiving vehicle sensor input and hazard determination
Receiving an environmental input from a sensor on a vehicle; determining a hazard state from the environmental input.
System reaction as an optimum temporal plan
Determining a system reaction comprising an optimum plan from a plurality of temporal plans.
Temporal plans as multi-step series of routes
Each temporal plan comprises a multi-step plan comprising a series of routes.
Incorporating user state into system reaction determination
Determining the system reaction from the hazard state and a user state; the user state determined from the environmental input in the form of an estimate of an expected user performance quality and an estimate of an expected user performance duration.
Using state estimation and predictive index
The user state is determined by correlating the environmental input to the user state utilizing a predictive index; the hazard state is determined by estimating the hazard state with a State Estimation Bayesian Network.
System reaction determination with uncertainty and automation
Determining the system reaction from the hazard state, a user state, an uncertainty model and an automation reaction.
Computer program product implementing temporal plan selection
A computer program product comprising code configured to receive environmental input from a sensor on a vehicle, determine a hazard state from the environmental input, determine a temporal plan from the hazard state, determine a system reaction from an optimum plan from a plurality of temporal plans where each temporal plan comprises a multi-step series of routes, and communicate the system reaction to a user interface.
The independent claims cover computer-implemented reception of environmental inputs from vehicle sensors, estimation of hazard and user (workload) states, determination of automation reactions from automation and user effectiveness, selection of system reactions as time-dependent temporal plans (multi-step series of routes) potentially optimized among alternatives, incorporation of uncertainty models and Bayesian and predictive-index based state estimation, and communicating the selected system reaction to multimodal user interfaces; a corresponding computer program product is claimed.
Stated Advantages
Increase the safety and efficiency of air traffic.
Provide improved flight deck displays to support pilots making more complex, strategic decisions in NextGen operations.
Improve the effectiveness of joint human-automation systems in aviation by adapting information presentation based on situational context.
Enable enhanced spatial awareness for flight crews, particularly with respect to separation from other aircraft and avoiding weather hazards.
Reduce information processing bottlenecks by displaying the right information at the right time in the right format through situationally-aware display adaptation and multimodal presentation.
ALARMS provides information acquisition support, enhances information integration and inference, and when appropriate provides decision support recommending trajectory or speed changes to divert around or avoid safety-critical hazards.
TBO-AID can address moving aircraft from Point A to Point B with greater efficiency, saving time, money, and fuel.
Documented Applications
Use with Next Generation Air Transportation System (NextGen) to support pilots responsible for following 4D trajectories and self-separation.
Integrated Alerting and Notification (IAN) system and the ALerting And Reasoning Management System (ALARMS) for flight deck alarming and decision support.
Trajectory Based Operations Adaptive Information Display (TBO-AID) as an adaptive flight deck display and multimodal user interface for pilots.
Application to vehicle operators such as car drivers.
Application to financial portfolio management.
Application to facility management.
Application to automated manufacturing processes.
Enhancement of collision avoidance systems such as the system described in U.S. Pat. No. 7,245,231 (Kiefer) by incorporating user state and temporal planning.
Interested in licensing this patent?