Method for determining an uncertainty level of deep reinforcement learning network and device implementing such method
Inventors
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12036391-B2
Publication Date
2024-07-16
Expiration Date
Abstract
Method for determining a recommendation value of a control parameter of a fluid infusion device. The method being implemented by a control device and comprising the steps of retrieving user data, feeding a deep reinforcement learning network, outputting a deep reinforcement learning network result, feeding an uncertainty certificates, outputting an uncertainty certificates result, comparing the uncertainty certificates result, determining the recommendation value, of a control parameter of the fluid infusion device based on a state of the unique user using a control algorithm or the deep reinforcement learning network.
Core Innovation
The invention provides a method for determining a recommendation value of a control parameter of a fluid infusion device by using a deep reinforcement learning network and uncertainty certificates. The method retrieves user data having a timestamp and being related to a unique user, where the user data comprises amounts of a drug infused, physiological values, and estimated values that represent a state of the unique user. The state is fed as input to a deep reinforcement learning network that comprises at least two layers.
The method outputs a deep reinforcement learning network result for at least a penultimate layer of the at least two layers. This penultimate-layer result is fed as an input to uncertainty certificates, which output an uncertainty certificate result. The uncertainty certificate result is compared to an uncertainty threshold value that represents a value under which the deep reinforcement learning network output is considered certain and otherwise considered uncertain.
If the deep reinforcement learning network output is considered certain, the recommendation value is determined based on the state using the deep reinforcement learning network. If the deep reinforcement learning network output is considered uncertain, the recommendation value is determined based on the state using another control algorithm. The uncertainty certificates comprise a set of functions of the state that are trained according to an explicit loss formulation including a trade-off parameter controlling the trade off between proximity of a plurality of certificates to the data and orthogonality of the certificates.
Claims Coverage
The independent claims cover three inventive features.
Recommendation value using DRL with uncertainty certificates
Determining a recommendation value of a control parameter of a fluid infusion device by retrieving timestamped user data for a unique user to form a state; feeding the state to a deep reinforcement learning network with at least two layers; outputting a deep reinforcement learning network result for at least a penultimate layer; feeding the penultimate-layer result to uncertainty certificates; comparing the uncertainty certificate result to an uncertainty threshold value; determining the recommendation value based on the DRL when considered certain, and based on the state using another control algorithm when considered uncertain.
Uncertainty determination for a trained DRL output using penultimate-layer certificates
Determining an uncertainty of a deep reinforcement learning network output of an already trained deep reinforcement learning network comprising at least two layers; retrieving input data comprising at least a state related to a process controlled by the device; feeding the state to the deep reinforcement learning network; outputting the deep reinforcement learning network result for at least a penultimate layer; feeding the penultimate-layer result to uncertainty certificates; comparing the uncertainty certificate result to an uncertainty threshold value; determining the recommendation value based on the state using the deep reinforcement learning network when considered certain and determining the recommendation value based on the state using another control algorithm when not considered certain; where the uncertainty certificates comprise a set of functions of the state trained according to a loss expression including a trade-off parameter between proximity to data and orthogonality of certificates.
Control device retrieving user state and using DRL with uncertainty certificates and fallback control algorithm
A control device for determining a recommendation value of a control parameter of a fluid infusion device comprising a retrieving unit configured to retrieve user data having a timestamp and related to a unique user, where the user data comprises amounts of a drug infused, physiological values, and estimated values representing a state; and a recommendation unit configured to determine the recommendation value based at least on the state by feeding the state to a deep reinforcement learning network with at least two layers; outputting a deep reinforcement learning network result for at least a penultimate layer; feeding an uncertainty certificates with the penultimate-layer result; comparing the uncertainty certificate result to an uncertainty threshold value; determining the recommendation value based on the state using the deep reinforcement learning network if considered certain; and determining the recommendation value based on the state using a control algorithm if not considered certain.
The claims center on penultimate-layer output of a deep reinforcement learning network being evaluated by uncertainty certificates, comparison to an uncertainty threshold to select between a DRL-derived recommendation and a control algorithm, and training the uncertainty certificates with a loss formulation that balances proximity to data and orthogonality.
Stated Advantages
Not explicitly described in patent.
Documented Applications
Not explicitly described in patent.
Interested in licensing this patent?