Performing inference using an adaptive, hybrid local/remote technique
Inventors
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11941503-B2
Publication Date
2024-03-26
Expiration Date
Abstract
A hybrid inference facility receives a sequence of data items. For each data item, the facility: forwards the data item to a server; subjects it to a local machine learning model to produce a local inference result for the data item; and the local inference result to a queue; aggregates the inference results contained by the queue to obtain an output inference result; and removes the oldest inference result from the queue. The facility receives from the server cloud inference results each obtained by applying a server machine learning model to one of the data items forwarded to the server. For each received cloud inference result, the facility substitutes the cloud inference result in the queue for the local inference result for the same data item.
Core Innovation
The document describes an adaptive hybrid inference facility in which a computing device and a cloud server cooperatively process a sequence of data items, including ultrasound image frames and ultrasound echo data. For each ultrasound image received by the computing device, the computing device attributes an identifier, dispatches the ultrasound image to a server, and subjects the ultrasound image to a local machine learning model to obtain a local inference result. The local inference result is added to a queue with the corresponding identifier, and the queue contents are aggregated to obtain an output inference result.
The facility aggregates inference results contained by the queue, removes the oldest inference result from the queue, and augments the ultrasound image with visual information derived from the output inference result for display. The computing device also receives from the server a plurality of cloud inference results, each obtained by applying a server machine learning model to an ultrasound image dispatched to the server, with an identifier indicating the source ultrasound image. For each received cloud inference result, the computing device substitutes the cloud inference result in the queue for the local inference result having the same identifier.
The document further describes adaptation to varying server/network conditions by buffering and by changing how many cloud inference results are available in the buffer at a given time. It contrasts local versus cloud models by describing configurations where the local machine learning model is designed to be evaluated using lesser computational resources than the cloud inference model, including local usage of reduced ensembles or simplified or selected models and a round-robin pseudo-ensemble approach using heterogeneous constituent models.
Claims Coverage
The partial content provides three independent claims that cover a queue-based hybrid inference workflow, a computer-readable medium performing the same processing, and a method configuring hybrid inference with cloud and local software/models and a circular buffer.
Ultrasound hybrid inference with queue aggregation and cloud substitution
An ultrasound system with an ultrasound sensing device and a computing device that receives ultrasound echo data comprising a sequence of ultrasound images, attributes an identifier to each ultrasound image, dispatches each ultrasound image to a server, subjects each ultrasound image to a local machine learning model to obtain a local inference result, adds each local inference result to a queue with its identifier, aggregates inference results in the queue to obtain an output inference result, removes the oldest inference result from the queue, augments the ultrasound image with visual information derived from the output inference result, receives a plurality of cloud inference results from the server with corresponding identifiers, and substitutes each received cloud inference result in the queue for the local inference result having the same identifier for display.
Hybrid inference via computer-readable media with queue substitution of cloud results
Computer-readable media configured to receive a sequence of data items, forward each data item to a server, subject the data item to a local machine learning model to produce a local inference result, add the local inference result to a queue, aggregate the inference results in the queue to obtain an output inference result, remove the oldest inference result from the queue, receive a plurality of cloud inference results from the server, and substitute each received cloud inference result in the queue for the local inference result for the same data item.
Configuring hybrid inference using cloud and lesser-resource local models with a circular buffer
A method for configuring hybrid inference for a sequence of data items generated by a local device by installing on a cloud server a cloud inference model and server software to receive data items from the local device, apply the cloud inference model, and return cloud inference results to the local device as obtained, and installing on the local device a local inference model evaluable using lesser computational resources than needed for the cloud inference model and local software to access data items as generated, send each data item to the cloud server, subject each data item to the local inference model to obtain and store local inference results in a circular buffer, aggregate inference results in the circular buffer, receive cloud inference results, and substitute each received cloud inference result in the circular buffer for the local inference result for the same data item.
Across the independent claims, the core coverage is a queue or circular-buffer-based hybrid inference workflow: local inference is performed per item, results are aggregated while removing the oldest entry, cloud inference results are returned with identifiers, and cloud results are substituted into the corresponding queue entry for the same item.
Stated Advantages
Documented Applications
Ultrasound sensing and processing of ultrasound echo data comprising a sequence of ultrasound images, with augmented ultrasound images displayed based on output inference results using hybrid local and cloud inference.
Interested in licensing this patent?