Systems and methods to label structures of interest in tissue slide images
Inventors
Zeineh, Jack • Prastawa, Marcel • Fernandez, Gerardo
Assignees
Icahn School of Medicine at Mount Sinai
Publication Number
US-11972621-B2
Publication Date
2024-04-30
Expiration Date
2040-04-14
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
Systems and methods to label structures of interest in tissue slide images are described.
Core Innovation
The invention relates to systems and methods to label structures of interest in tissue slide images, specifically to improve the efficiency and accuracy of generating labels for biomedical images compared to conventional techniques. It provides a computerized approach to process scanned images of tissue slides that have been stained with different stains but share a common attribute, enabling better identification and annotation of regions of interest.
The problem being addressed arises from the laborious, error-prone, and difficult-to-reproduce nature of manual labeling of tissue slide images, which limits the ability to generate large quantities of labeled data necessary for machine learning systems. Manual tracing to delineate structures such as epithelial and stromal regions is time-consuming and prone to inaccuracies especially in defining epithelial boundaries, hindering efficient biomedical image analysis.
The disclosed system involves scanning stained tissue slides with different stains (such as hematoxylin and eosin and immunohistochemistry markers), preprocessing these images by separating foreground and background and unmixing color channels, and performing a two-stage image registration based on a common stain attribute (e.g., hematoxylin channel). It further identifies candidate regions of interest either manually or automatically using heuristics or machine learning techniques, generates label annotations including segmentation masks for structures of interest, and integrates human expert review through a graphical user interface to update labels and comments, which can then be used as training data for machine learning models.
Claims Coverage
The patent includes three independent claims directed to a computer-implemented method, a system, and a computer readable medium, each comprising inventive features related to labeling structures in tissue slide images.
Two-stage image registration using a common stain attribute
Performing image registration on tissue slide images using a common attribute image (hematoxylin channel) by minimizing a mutual information metric in two stages: low-resolution registration with global rigid and affine transforms, followed by high-resolution registration of localized fields using affine and B-spline transforms.
Preprocessing by foreground-background separation and color unmixing
Separating foreground structures from background and unmixing color to separate components of stains into different channels, including separating hematoxylin and eosin channels for the first stain and separating hematoxylin and DAB channels for the second stain.
Identifying candidate regions of interest using prelabeling and sampling
Identifying candidate regions by prelabeling tumor areas, randomly sampling inside and outside the tumor, and for phosphohistone-stained slides, identifying regions with highest density of PHH3.
Aligning slide images for the same tissue area
Creating a registered image that includes the first and second tissue slide images aligned for the same tissue area to facilitate labeling and analysis.
Providing images and annotations through graphical user interface (GUI) for review and updating
Providing tissue slide images and label annotations to a reviewer system GUI, receiving updated labels or comments from reviewers, storing updated information with images, and supplying labeled images as training data to machine learning models.
The claims collectively cover a comprehensive system and method for labeling structures in tissue slide images, including preprocessing of images, sophisticated two-stage registration based on common stain attributes, candidate region identification with sampling techniques, GUI-based expert review for label refinement, and integration with machine learning training pipelines.
Stated Advantages
More accurate and efficient generation of labels compared to conventional manual techniques.
Reduction of labor intensity and error-proneness in generating labeled training data for machine learning.
Facilitation of interactive and incremental learning through near real-time integration of annotated images and expert feedback.
Improved alignment and correspondence between different stained images enabling better localization and labeling of tissue structures.
Documented Applications
Generating labeled images as training data for machine learning models that identify structures of interest in tissue slide images.
Biomedical image processing for cancer diagnosis and research, exemplified in breast tissue analysis but extendable to other tissue types and cancer sites.
Supporting expert pathologists and non-clinicians in annotating mitotic figures and epithelial regions using graphical user interfaces that overlay registered images and label masks.
Automated and manual region of interest identification and annotation in tissue histology slides stained with hematoxylin and eosin and immunohistochemistry markers such as PHH3 and CK18.
Interested in licensing this patent?