Systems and methods to label structures of interest in tissue slide images
Inventors
Zeineh, Jack • Prastawa, Marcel • Fernandez, Gerardo
Assignees
Icahn School of Medicine at Mount Sinai
Publication Number
US-12243335-B2
Publication Date
2025-03-04
Expiration Date
2040-04-14
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
Systems and methods to label structures of interest in tissue slide images are described.
Core Innovation
The invention relates to systems and methods to label structures of interest in tissue slide images, specifically improving accuracy and efficiency over conventional labeling techniques. The methods involve generating tissue slide images stained with different stains, preprocessing to separate foreground from background structures and to separate stain components into color channels, and performing image registration using a common attribute of the stains, such as the hematoxylin channel. Subsequently, candidate regions of interest in the images are identified for labeling structures.
The registration process is performed in two stages optimizing a mutual information metric between the hematoxylin channels, beginning with coarse, low-resolution registration and followed by high-resolution registration using affine and B-spline transforms. The system includes generating label annotations for the structures in the registered tissue slide images and optionally receiving manual annotation updates via a graphical user interface, which can then be used as training data for machine learning models, particularly convolutional neural networks.
The background identifies the problem that manual generation of labeled data in tissue imaging is tedious, labor intensive, prone to error, and not reproducible due to the large data volumes. Conventional methods require manual tracing to delineate structures such as epithelial and stromal regions, which is difficult to perform accurately. This invention addresses these challenges by automating and improving the labeling process with computational image processing and registration techniques, thereby facilitating efficient and accurate creation of labeled datasets necessary for training machine learning models in biomedical imaging.
Claims Coverage
The patent includes a set of independent claims covering methods, systems, and computer-readable media for labeling structures in tissue slide images. The main inventive features relate to image registration processes, candidate region identification, mask generation, and integration with machine learning models.
Two-stage image registration method for tissue slide images
Performing image registration by minimizing a mutual information metric between deconvolved hematoxylin channels from slides stained with two different stains. The registration method comprises two stages: (1) coarse, low-resolution registration using global rigid and affine transforms, and (2) high-resolution registration of localized fields using affine and B-spline transforms.
Identification of candidate regions of interest from registered tissue slide images
Based on the image registration, identifying one or more candidate regions of interest (ROIs) within the plurality of tissue slide images for labeling structures of interest.
Generation of label annotations for tissue slide structures
Generating one or more label annotations for structures within the tissue slide images based on the registered images and identified ROIs.
Generating segmentation masks including epi-stroma masks
Creating an epi-stroma mask that identifies epithelium and stroma regions by creating an initial mask estimating epithelial cells and separating background from tissue, extracting image features, clustering image pixels based on these features, and generating the epi-stroma mask from clustered pixels.
Receiving manual annotation updates and training CNN models
Receiving manual annotation updates from a reviewer system graphical user interface and providing these manual updates as training data for a convolutional neural network (CNN) model, which can be trained to output mitotic figure candidates.
Extraction of hierarchical ROIs from hematoxylin and eosin and immunohistochemistry images
Identifying a hematoxylin and eosin ROI and extracting a first immunohistochemistry ROI based on the hematoxylin and eosin ROI, extracting a common channel from both ROIs, and after registration, extracting a smaller second immunohistochemistry ROI corresponding to the hematoxylin and eosin ROI.
The claims cover computer-implemented methods, systems, and computer-readable media implementing a two-stage mutual information registration process, automated ROI identification, mask generation for epithelium and stroma, annotation updates via GUI, and integration of labeled data into training convolutional neural networks for tissue structure labeling.
Stated Advantages
Provides more accurate and efficient generation of labels for tissue slide images compared to conventional manual techniques.
Reduces labor intensity and improves reproducibility by automating region identification and registration using image processing.
Enables integration with machine learning models to improve automated detection and labeling of structures such as mitotic figures and epithelium.
Facilitates an efficient human review process with interactive graphical user interfaces that allow quick switching between registered H&E and IHC images for review and annotation.
Improves training of machine learning models by providing corrected and validated labeled images from expert reviewers.
Documented Applications
Biomedical image processing for labeling structures of interest in tissue slide images, such as identifying epithelium, stroma, and mitotic figures in stained tissue samples.
Generating training data for machine learning models to automate detection and classification of cellular structures in histological slides.
Improvement of pathology workflows by supporting experts in annotation and review of tissue samples using enhanced GUI tools.
Application in cancer diagnosis, exemplified with breast tissue analysis, and extendable to other cancer sites and biomedical imaging purposes.
Interested in licensing this patent?