Modality-agnostic method for medical image representation
Inventors
Sjölund, Jens Olof • Adler, Jonas Anders
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-10762398-B2
Publication Date
2020-09-01
Expiration Date
Abstract
Techniques for the operation and use of a model that learns the general representation of multimodal images is disclosed. In various examples, methods from representation learning are used to find a common basis for representation of medical images. These include aspects of encoding, fusion, and downstream tasks, with use of the general representation and model. In an example, a method for generating a modality-agnostic model includes receiving imaging data, encoding the imaging data by mapping data to a latent representation, fusing the encoded data to conserve latent variables corresponding to the latent representation, and training a model using the latent representation. In an example, a method for processing imaging data using a trained modality-agnostic model includes receiving imaging data, encoding the data to the defined encoding, processing the encoded data with a trained model, and performing imaging processing operations based on output of the trained model.
Core Innovation
The invention provides a computer implemented method for generating a modality-agnostic image processing model for medical imaging processing. The method receives imaging data from multiple imaging modalities and applies a defined data encoding for each modality, mapping the imaging data to respective latent representations and producing respective latent variables corresponding to a spatial representation.
The method fuses the encoded imaging data by mapping the respective latent representations to a fused latent representation of the encoded imaging data using the respective latent representations and the respective latent variables. The mapping combines the respective latent representations into the fused latent representation while conserving structure indicated by the respective latent variables.
The fused latent representation is used for training a model for medical imaging processing, and the trained modality-agnostic model is adapted to produce an output from subsequent medical imaging data according to the training using the fused latent representation. A modality-agnostic imaging processing system is also provided, including processing circuitry and storage medium instructions that cause the processor to process imaging data, perform the encoding and mapping into a common latent representation, train the model using the common latent representation, and utilize the trained model to produce outputs for an image processing operation.
Claims Coverage
The document includes three independent claims. Across the claims, there are 3 core inventive feature groupings: latent-variable spatial encoding per modality, structure-conserving fusion into a fused/common latent representation, and training and using a downstream medical imaging processing model based on the fused/common latent representation.
Modality-agnostic model generation via spatial latent encoding and structure-conserving fusion
Receiving imaging data from multiple imaging modalities; encoding the imaging data by mapping the imaging data to respective latent representations and producing respective latent variables corresponding to a spatial representation; fusing by mapping the respective latent representations to a fused latent representation using the latent representations and the latent variables, combining the respective latent representations into the fused latent representation while conserving structure indicated by the respective latent variables; training a model for medical imaging processing using the fused latent representation; and outputting the trained model adapted to produce an output from subsequent medical imaging data according to the training using the fused latent representation.
Processing imaging data with a trained modality-agnostic model using fused latent representation training
Receiving medical imaging data of a first imaging modality; encoding the medical imaging data to a latent representation based on a defined data encoding; processing the encoded medical imaging data with a trained model to produce an output based on the latent representation, wherein training of the model is performed using mapping of training imaging data to respective latent representations and producing respective latent variables corresponding to a spatial representation; fusing the encoded training imaging data by mapping the respective latent representations to a fused latent representation while conserving structure indicated by the respective latent variables; and performing an image processing operation on the medical imaging data based on the output produced by the trained model.
System with encoding, structure-conserving mapping to a common latent representation, and trained-model output
A system comprising processing circuitry and storage medium instructions causing the processor to: process imaging data produced from a medical imaging modality; encode the imaging data to a latent representation based on a defined data encoding with one or more latent variables corresponding to a spatial representation; perform mapping of the encoded imaging data to a common latent representation using the latent representation and the one or more latent variables while conserving structure indicated by the one or more latent variables; train a model for medical imaging processing using the common latent representation; and utilize the trained model to produce an output from subsequent medical imaging data in connection with an image processing operation.
Across the independent claims, the inventive coverage is directed to modality-agnostic medical imaging representation learning that encodes multiple modality inputs into latent representations with latent variables corresponding to spatial representation, maps and fuses those representations into a fused or common latent representation while conserving structure indicated by the latent variables, and trains and uses a medical imaging processing model so that outputs can be produced for subsequent medical imaging data.
Stated Advantages
Not explicitly described in patent.
Documented Applications
Radiotherapy planning operations using the output of the trained model.
Segmentation, denoising, synthesis, classification, regression, and reconstruction as image processing operations.
Interested in licensing this patent?