Systems and methods for frame-based validation
Inventors
Pokkalla, Harsha Vardhan • Elliott, Hunter L. • Wang, Dayong • Glass, Benjamin P. • Wapinski, Ilan N. • Kerner, Jennifer K. • Beck, Andrew H. • Khosla, Aditya • Gullapally, Sai Chowdary • Srinivasan, Ramprakash
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11915823-B1
Publication Date
2024-02-27
Expiration Date
Abstract
In some aspects, the described systems and methods provide for validating performance of a model trained on a plurality of annotated pathology images. A pathology image is accessed. Frames are generated using the pathology image. Each frame in the set includes a distinct portion of the pathology image. Reference annotations are received from one or more users. The reference annotations describe at least one of a plurality of tissue or cellular characteristic categories for one or more frames in the set. Each frame in the set is processed using the trained model to generate model predictions. The model predictions describe at least one of the tissue or cellular characteristic categories for the processed frame. Performance of the trained model is validated based on determining a degree of association between the reference annotations and the model predictions for each frame and/or across all frames in the set of frames.
Core Innovation
The invention relates to a frame-based validation framework for a trained model configured to predict at least one of a plurality of tissue and/or cellular characteristic categories from a pathology image. A set of frames is generated such that each frame includes a portion of a pathology image, and reference annotations describing at least one tissue and/or cellular characteristic category are received from a plurality of users for the one or more frames.
The trained model processes the set of frames to generate model predictions that produce predicted annotations, where each predicted annotation describes at least one of the plurality of tissue and/or cellular characteristic categories for a processed frame. Performance is validated by determining a degree of association between the reference annotations and the model predicted annotations for the set of frames.
The framework supports multiple ways to quantify the degree of association, including consensus clusters using spatial proximity, concordance-based evaluation that compares user reference annotations within a user cluster, correlations between consensus scores and model scores, and performance metrics such as true positive rate, precision, recall, sensitivity, specificity, mean Average Precision (mAP), and mean Average Recall (mAR) over a hyperparameter sweep. The approach is motivated by pathologist inter-/intra-observer variability and supports high-quality ground truth and scalable quantitative measurements, including PD-L1.
Claims Coverage
The independent claims cover validating a trained model by receiving reference annotations from a plurality of users for frames from pathology images, generating model predictions for those frames, and determining a degree of association between reference annotations and model predicted annotations across the set of frames. The inventive coverage includes method, system, and non-transitory storage medium implementations.
Frame-based association validation of pathology-image model predictions
Receiving, from a plurality of users, reference annotations for one or more frames in a set of frames for one or more pathology images, processing the set of frames using the trained model to generate model predictions of annotations, and validating performance based on determining a degree of association between the reference annotations and the model predicted annotations for the set of frames.
System for association-based performance validation using plurality of user reference annotations
A system comprising at least one computer hardware processor and at least one non-transitory computer-readable storage medium storing processor-executable instructions to receive reference annotations for frames, process the set of frames using the trained model to generate model predictions of annotations, and validate performance by determining a degree of association between the reference annotations and the model predicted annotations.
Non-transitory storage medium for association-based performance validation
At least one non-transitory computer-readable storage medium storing processor-executable instructions that cause a processor to perform receiving reference annotations, processing the set of frames using the trained model to generate model predictions of annotations, and validating performance by determining a degree of association between the reference annotations and the model predicted annotations.
Concordance aggregation within user clusters to determine degree of association
Determining the degree of association by computing concordance based on comparing reference annotations within a user cluster and comparing model predicted annotations versus reference annotations within that user cluster, and aggregating concordances across the plurality of users to determine the overall degree of association.
Performance-metric instantiation of the degree of association
Validating the degree of association using performance metrics including true positive rate, precision, recall, sensitivity, specificity, mean Average Precision (mAP), and mean Average Recall (mAR) across a hyperparameter sweep of the trained model.
The core validation concept is determining a degree of association between plurality-of-user reference annotations and trained-model predicted annotations on sets of pathology-image frames. Dependent refinements specify concordance within user clusters with aggregation and performance metrics such as mAP and mAR over a hyperparameter sweep.
Stated Advantages
Provides high-quality ground truth and scalable quantitative measurements for validating pathology-image model performance.
Enables quantitative evaluation tied to consensus and user/pathologist variability (inter-/intra-observer variability).
Supports measurable validation outcomes for categories including PD-L1.
Documented Applications
Validating a pathology-image model for predicting tissue and/or cellular characteristic categories using crowd-sourced reference annotations, including PD-L1 immunohistochemistry (IHC).
Interested in licensing this patent?