Progressive and multi-path holistically nested networks for segmentation
Inventors
Harrison, Adam Patrick • Xu, Ziyue • Lu, Le • Summers, Ronald M. • Mollura, Daniel Joseph
Assignees
US Department of Health and Human Services
Publication Number
US-11195280-B2
Publication Date
2021-12-07
Expiration Date
2038-06-08
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
Methods include processing image data through a plurality of network stages of a progressively holistically nested convolutional neural network, wherein the processing the image data includes producing a side output from a network stage m, of the network stages, where m>1, based on a progressive combination of an activation output from the network stage m and an activation output from a preceding stage m−1. Image segmentations are produced. Systems include a 3D imaging system operable to obtain 3D imaging data for a patient including a target anatomical body, and a computing system comprising a processor, memory, and software, the computing system operable to process the 3D imaging data through a plurality of progressively holistically nested convolutional neural network stages of a convolutional neural network.
Core Innovation
The invention disclosed relates to systems and methods for progressive and multi-path holistically nested convolutional neural networks (P-HNNs) for image segmentation, including segmentation of pathological lungs, organs, tumors, or other bodies from CT images, as well as objects from natural images. The technology enhances standard holistically nested networks (HNNs) by introducing a progressive combination of activation outputs from multiple sequential network stages to produce progressively refined segmentation masks.
The disclosed methods address limitations in existing pathological lung segmentation (PLS) approaches, which are often complex, lack generality, or fail to produce sufficiently accurate or reliable segmentations due to wide variability in pathological appearance and shape. The invention employs a bottom-up deep-learning framework with deep supervision and a progressive multi-path scheme that merges outputs from different network stages more reliably, producing finer detailed masks without additional network parameters, thereby improving usability and performance.
Claims Coverage
The patent includes multiple independent claims covering methods and systems that utilize progressively holistically nested convolutional neural networks for image segmentation. There are three main independent claims directed to a method, a computing system, and a system including a 3D imaging system and computing system.
Processing image data through progressively holistically nested convolutional neural networks
The method processes image data through multiple network stages where a side output from a network stage m (m>1) is produced based on a progressive combination of activation outputs from the current stage m and the preceding stage m−1, enhancing segmentation output progressively.
Incorporation of progressive constraints on multi-scale pathways without additional parameters
The progressive combination of activation outputs includes a simple addition operation, applying a progressive constraint on multi-scale pathways that improves segmentation results with no extra convolutional layers or network parameters required.
Training through deep supervision at each network stage with class-balancing
The convolutional neural network is trained via deep supervision at each stage by processing multiple input training images with known ground truth segmentations, determining cross-entropy loss using a class-balancing weight derived from average image segmentation edge ground truths, then backpropagating with gradient descent to update network parameters.
System including 3D imaging and computing system operable to process image data through P-HNN stages
A system that includes a 3D imaging device (e.g., CT scanner) operable to obtain 3D imaging data of a patient with a target anatomical body, and a computing system that processes this data through multiple progressively holistically nested convolutional neural network stages producing side outputs based on progressive combinations and generating final image segmentations accordingly.
The claims collectively cover innovative methods and systems that utilize progressive combinations of activation outputs in holistically nested convolutional neural networks for improved image segmentation, training procedures with deep supervision and class balancing, and integration with 3D imaging systems to segment anatomical or non-anatomical targets with enhanced accuracy and efficiency.
Stated Advantages
The progressive multi-path enhancement allows producing finer detailed segmentation masks with improved accuracy over standard holistically nested networks.
The approach requires no additional network parameters, maintaining methodological simplicity and generality important for usability.
The technology can reliably merge outputs from different network stages to provide consistent segmentation outputs, avoiding inconsistencies found in standard HNN fused outputs.
The method successfully handles wide variability in pathological appearance and shape, producing robust pathological lung segmentations.
The system improves convergence during network training by including batch normalization layers, which reduces training time and variance between training sets.
The approach effectively functions on 2D CT images from 3D volumes, overcoming limitations of 3D CNNs, such as field-of-view constraints and inter-slice discontinuities.
Documented Applications
Segmentation of pathological lungs from CT images, including lungs with infection, interstitial lung disease (ILD), or chronic obstructive pulmonary disease (COPD).
Segmentation of other anatomical organs, tumors, or bodies from 3D medical imaging data such as CT or MRI scans.
Segmentation of non-anatomical objects from natural images.
Integration with 3D imaging systems (e.g., computerized tomography systems) for clinical imaging data processing to produce anatomical segmentation masks usable in physiological measurements such as lung volume assessment.
Interested in licensing this patent?