Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Assignees
Leland Stanford Junior University
MemberStanford UniversityStanford UniversityTodd Coleman is an associate professor of bioengineering at Stanford University. His research combines applied probability, physiology, and bioelectronics to develop methods and devices for monitoring and modulating nervous-system and other physiological activity, including brain and visceral-organ signals.
Todd Coleman is an associate professor of bioengineering at Stanford University. His research combines applied probability, physiology, and bioelectronics to develop methods and devices for monitoring and modulating nervous-system and other physiological activity, including brain and visceral-organ signals.
Publication Number
US-12737675-B2
Publication Date
2026-09-15
Expiration Date
Abstract
Systems and methods for imitation learning in accordance with embodiments of the invention are illustrated. One embodiment includes a method for imitation learning. The method includes steps for initializing a Q-function, training the Q-function using a non-adversarial objective based on a set of one or more expert trajectories, and determining a policy based on the trained Q-function.
Core Innovation
IQ-Learn is a dynamics-aware imitation-learning framework that learns a single Q-function from expert trajectories using a non-adversarial objective, rather than adversarial joint reward and policy optimization. The Q-function implicitly represents both policy and reward. A policy is derived from the Q-function or iteratively updated with it using actor-critic or soft actor-critic methods.
The framework defines recovered transition rewards using the Q-function and the policy's state-value function, and uses regularizers to support statistical-distance objectives including total variation, Wasserstein, and χ² divergence. The supplied partial content describes reported benchmark performance, faster convergence, and recovered rewards correlated with ground truth, but does not include the background section identifying the problem being solved.
Claims Coverage
The three independent claims are claims 1, 13, and 19. They cover three inventive features: a method for imitation learning, a system utilizing an imitation-learning model to control operation, and a non-transitory machine-readable medium containing processor instructions for imitation learning.
Training a Q-function and iteratively updating it with a policy
Claim 1 recites training, with a processor, a Q-function using a non-adversarial objective based on one or more expert trajectories; determining a policy for controlling an output device based on the trained Q-function; and iteratively updating the trained Q-function based on the determined policy and the determined policy based on the trained Q-function.
Controlling an output device using an imitation-learning model
Claim 13 recites a processor and memory containing a control application that obtains current state information of an output device, provides that information to an imitation-learning model using a single Q-function, and obtains control data from the model based on a final version of the determined policy. The model trains the single Q-function using a non-adversarial objective based on expert trajectories and iteratively updates the Q-function and policy.
Processor instructions for imitation learning
Claim 19 recites a non-transitory machine-readable medium containing processor instructions that cause a processor to initialize and train a Q-function using a non-adversarial objective based on one or more expert trajectories, determine a policy for controlling an output device based on the trained Q-function, and iteratively update the Q-function and policy based on each other.
The independent claims cover method, system, and machine-readable-medium forms of imitation learning based on training a Q-function from expert trajectories with a non-adversarial objective and iteratively updating the Q-function and policy. The system claim also recites using the model to obtain control data for an output device.
Stated Advantages
The disclosure reports strong benchmark performance, including expert-level results in some cases from one demonstration.
The framework is reported to converge faster.
Recovered rewards are reported to correlate with ground-truth rewards.
Documented Applications
Using the determined policy to control an output device. The claims identify medical devices, video game devices, robots, and autonomous vehicles as output-device examples.
Using the determined policy to drive an artificial intelligence (AI) bot.
Evaluating policies using recovered rewards.
Interested in licensing this patent?