Synthetic scaling applied to shared neural networks
Inventors
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11669587-B2
Publication Date
2023-06-06
Expiration Date
Abstract
A system processing a stream of input data ordered row by row from a data array has a first integrated circuit (IC) adapted to apply an aperture function to the stream of input data, to produce an output data stream, and a second IC coupled to the first IC, the second IC adapted to manage context from row to row, retaining partial values as computed by the aperture function for each column along a row, and providing the partial values back to the aperture function for subsequent rows as needed to complete output values.
Core Innovation
A pipelined system processes a stream of input data ordered column by column and row by row from a data array. A first integrated circuit simultaneously multiplies each datum as received by every weight value of a specific aperture function and applies a subfunction of the specific aperture function to each datum as received. The first IC produces partial values of the specific aperture function for each datum and produces an output value at an output port for each column and row position.
A second IC manages and retains the partial values for each column along each row. The second IC passes the partial values through a connected series of first-in-first-out and/or shift registers and provides the partial values back to the first IC for subsequent rows as needed to develop and complete output values. The output values are produced as a summation of the partial values for each input datum for a patch position of the specific aperture function in order, and the system produces a stream of output values synchronously with the stream of input values.
The disclosed architecture extends the pipelined aperture processing to N-up parallel CNN/DNN streaming and to 3D aperture processing with 3D compositors/FIFOs buffering partial context. A further extension time-shares a single IC aperture function across multiple input sources or multiple dynamically generated scales using context switching and syncopated interleaving, producing a multi-scale interleaved output stream.
Claims Coverage
The independent claim defines a pipelined hardware system with two ICs that generate aperture-function summations from streamed input data while synchronously producing a stream of output values. Dependent claims further refine input interleaving across multiple streams, scaling by a fixed ratio with interleaving, and mapping the interleaved stream processing onto nodes of a convoluted neural network (CNN) without further downscaling.
Pipelined aperture-function summation from streamed data
A first integrated circuit simultaneously multiplies each datum as received by every weight value of a specific aperture function and applies a subfunction of the specific aperture function to each datum of the stream, producing a partial value for each datum and producing an output value for each column and row position.
FIFO/shift-register partial-value retention and feedback
A second IC coupled to the first IC receives and manages the partial values for each column along each row, passing the partial values through a connected series of first-in-first-out and/or shift registers and providing the partial values back to the first IC for subsequent rows as needed to develop and complete output values.
Synchronous output stream as ordered summation of retained partial values
The output values are a summation of the partial values produced for each input datum for a patch position of the specific aperture function at each application position of the patch in order, and the system produces a stream of output values synchronously with the stream of input values.
Interleaving IC for combining first and second input streams
First and second input data streams from respective data arrays, and an interleaving integrated circuit that interleaves them into a single interleaved data stream supplied to the first IC.
Sampler IC circuit for fixed-ratio scaling and interleaved multi-scale output
A sampler IC circuit that receives input data, scales it by a fixed ratio, and uses an interleaving IC to interleave a full-scale stream and a scaled stream into an interleaved multi-scale output data stream.
CNN node interleaving without further downscaling
An interleaved data stream from a first node of a convoluted neural network is interleaved by a first IC and then processed as an interleaved output stream in a second CNN node without further downscaling.
Across the independent and dependent claims, the inventive concept is a two-IC pipelined aperture-function computation over streamed, ordered input data, where partial values are generated via aperture-weighted multiplication and retained via FIFO/shift-register structures for feedback across subsequent rows to complete ordered summations. Additional claim refinements interleave multiple input streams, produce multi-scale interleaved output via fixed-ratio scaling, and align the interleaved stream processing to CNN nodes without further downscaling.
Stated Advantages
Produces a stream of output values synchronously with the stream of input values.
Retains and manages partial values to develop and complete output values for patch positions in order.
Enables interleaved multi-scale output data streams via fixed-ratio scaling and interleaving.
Supports CNN-node processing of interleaved streams without further downscaling.
Documented Applications
Mapping interleaving and processing onto nodes of a convoluted neural network (CNN) as interleaved output streams.
N-up parallel CNN/DNN streaming.
3D (volumetric) aperture processing, including 3D convolution over voxels.
Multi-scale interleaved output generation by time-sharing an IC aperture function across multiple input sources or multiple dynamically generated scales.
Interested in licensing this patent?