Progressive neural ordinary differential equations
Inventors
Yao, Yi • Divakaran, Ajay • Ayyubi, Hammad A.
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12602448-B2
Publication Date
2026-04-14
Expiration Date
Abstract
Techniques are described for neural networks based on Progressive Neural ODEs (PODEs). In an example, a method to progressively train a neural ordinary differential equation (NODE) model comprises processing, by a machine learning system executed by a computing system, first training data, the first training data having a first complexity, to perform training of a first layer for the NODE model; and after performing the first training, processing second training data, the second training data having a second complexity that is higher than the first complexity, to perform training of a second layer for the NODE model.
Core Innovation
Progressive Neural ODE (PODE) progressively trains a neural ordinary differential equation (NODE) model by separating training into a first stage and a second stage. The first stage processes first training data comprising irregularly spaced time series data derived from second training data by applying a complexity-reducing transformation that reduces frequency content of the second training data. The first stage trains a first layer of the NODE model, where the first layer comprises a first ODE-based transfer function parameterized by a first set of parameters.
After processing the first training data, the model retains the first set of parameters of the first layer fixed while the training proceeds to a second stage. In the second stage, the system processes the second training data comprising irregularly spaced time series data to perform training of a second layer for the NODE model. The second layer is different from the first layer and comprises a second ODE-based transfer function parameterized by a second set of parameters.
The document also describes an encoder–NODE–decoder architecture in which an encoder maps irregularly spaced time series data into fixed-length embedded representations, and the decoder outputs predicted values back as time-series output data that can be irregularly spaced. The progressive training is additionally described as being tied to time-series complexity defined using time-series properties such as trends, periodicity/seasonality, irregularity, and frequency content, including example complexity-reduction such as low-pass filtering or principal component analysis (PCA). The document also describes optional stabilization when adding layers using alpha blending controlled by a configurable parameter a.
Claims Coverage
Across the provided independent claims, the core claim coverage is a staged, layer-wise progressive training of a NODE model using irregularly spaced time-series data derived by a complexity-reducing transformation that reduces frequency content. The independent claims collectively specify an ODE-based transfer function parameterized by first and second parameter sets, with the first set retained fixed while the second layer is trained, and they extend this concept to computing-system execution, a method form, and non-transitory computer readable media instructions.
Complexity-reducing transformation for progressive NODE layer training
Process first training data comprising irregularly spaced time series data derived from second training data by applying a complexity-reducing transformation that reduces frequency content of the second training data to perform training of a first layer for the NODE model.
Retaining first layer parameters while training a second ODE-based layer
After processing the first training data to perform the training of the first layer for the NODE model, process while retaining the first set of parameters of the first layer fixed the second training data comprising irregularly spaced time series data to perform training of a second layer for the NODE model, the second layer different from the first layer and comprising a second ODE-based transfer function parameterized by a second set of parameters.
ODE-based transfer functions parameterized for first and second layers
The first layer comprises a first ordinary differential equation (ODE)-based transfer function parameterized by a first set of parameters, and the second layer comprises a second ODE-based transfer function parameterized by a second set of parameters.
Progressive training implemented as a computing system execution
A computing system comprising a machine learning system, memory configured to store the NODE model, and processing circuitry configured to execute the machine learning system to process first training data and second training data as the progressive training of the first layer and the second layer.
Progressive training implemented as a method
A method comprising processing first training data to train a first layer for the NODE model and, after processing the first training data, processing while retaining the first set of parameters fixed second training data to train a second layer for the NODE model.
Instructions on non-transitory computer readable media for progressive NODE layer training
Non-transitory computer readable media comprising instructions for causing processing circuitry to execute a machine learning system configured to process first training data and then process while retaining the first set of parameters fixed second training data to progressively train a first layer and a second layer of a NODE model.
The independent claims cover progressive, layer-wise training of a NODE model using irregularly spaced time-series data, where first-stage data is derived by a complexity-reducing transformation that reduces frequency content and the first layer’s parameters are retained fixed while training a different second ODE-based transfer function layer.
Stated Advantages
Reports experimental improvements over conventional/vanilla NODE, including synthetic MSE reduction and >64% improvement on PEMS-BAY traffic forecasting.
Documented Applications
Traffic forecasting on the PEMS-BAY dataset.
Use with irregularly sampled time series in an encoder–NODE–decoder architecture for prediction and output data.
Interested in licensing this patent?