Systems and methods for frame-based validation

Inventors

Pokkalla, Harsha Vardhan • Elliott, Hunter L. • Wang, Dayong • Glass, Benjamin P. • Wapinski, Ilan N. • Kerner, Jennifer K. • Beck, Andrew H. • Khosla, Aditya • Gullapally, Sai Chowdary • Srinivasan, Ramprakash

Assignees

PathAI Inc

Abstract

In some aspects, the described systems and methods provide for validating performance of a model trained on a plurality of annotated pathology images. A pathology image is accessed. Frames are generated using the pathology image. Each frame in the set includes a distinct portion of the pathology image. Reference annotations are received from one or more users. The reference annotations describe at least one of a plurality of tissue or cellular characteristic categories for one or more frames in the set. Each frame in the set is processed using the trained model to generate model predictions. The model predictions describe at least one of the tissue or cellular characteristic categories for the processed frame. Performance of the trained model is validated based on determining a degree of association between the reference annotations and the model predictions for each frame and/or across all frames in the set of frames.

Core Innovation

The invention provides a frame-based validation framework for performance of a trained model trained on a plurality of annotated pathology images. Each pathology image includes at least one annotation describing tissue or cellular characteristic categories for one or more portions of the image, and the method accesses a pathology image and generates a set of frames in which each frame includes a distinct portion of the pathology image.

The framework receives, from one or more users, reference annotations describing at least one of a plurality of tissue or cellular characteristic categories for one or more frames in the set of frames. The trained model processes each frame in the set of frames to generate model predictions describing at least one of the plurality of tissue or cellular characteristic categories for the processed frame. Performance is validated by determining a degree of association between the reference annotations and the model predictions for each frame and/or across all frames in the set of frames.

In further refinements, the degree of association can be based on consensus clustering to generate consensus clusters and consensus annotations, together with a measure of spatial proximity for associating model predictions to consensus clusters. Quantitative validation can include true positive rate, precision, recall, sensitivity, specificity, mean Average Precision (mAP), mean Average Recall (mAR), Pearson correlation coefficient, and intraclass correlation coefficient (ICC). Frame selection and validation can also incorporate feature value distributions and exclude frames with inadequate quality, tissue staining, and/or artifacts.

Claims Coverage

The independent claims define a framework for validating performance by frame generation and a degree of association between reference annotations and model predictions. Across the family, inventive features refine the association computation, evaluation metrics, data quality constraints, consensus clustering with spatial proximity, and exhaustive reference annotations from a plurality of pathologists.

Overall claim coverage centers on generating frame portions from annotated pathology images, collecting user reference annotations for tissue or cellular characteristic categories on frames, producing model predictions for the same categories, and validating performance via a degree of association between reference annotations and predictions, with optional refinements such as consensus clustering with spatial proximity, metric-based evaluation including mAP and mAR, and filtering out inadequate or artifact-containing frames.

Stated Advantages

Documented Applications