Augmented and mixed reality incorporating pathology results in surgical settings
Inventors
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Assignees
MemberUniversity of MiamiUniversity of MiamiThe University of Miami, established in 1925 and based in Coral Gables, Florida, is a private research university recognized for its comprehensive academic offerings, robust research infrastructure, and strong interdisciplinary focus. Home to more than 19,000 students and with more than 400 acres of campuses across the Miami region, its mission encompasses education, research, innovation, and community service. The institution supports clinical, biomedical, marine, and atmospheric research initiatives, delivers diverse undergraduate and graduate programs, and maintains numerous research centers and institutes dedicated to scientific, medical, and societal advancements.
The University of Miami, established in 1925 and based in Coral Gables, Florida, is a private research university recognized for its comprehensive academic offerings, robust research infrastructure, and strong interdisciplinary focus. Home to more than 19,000 students and with more than 400 acres of campuses across the Miami region, its mission encompasses education, research, innovation, and community service. The institution supports clinical, biomedical, marine, and atmospheric research initiatives, delivers diverse undergraduate and graduate programs, and maintains numerous research centers and institutes dedicated to scientific, medical, and societal advancements.
Publication Number
US-12714513-B2
Publication Date
2026-08-25
Expiration Date
Abstract
Apparatuses and methods for using augmented/mixed reality in surgical settings are provided, and, in particular, apparatuses and methods for intraoperative integration of augmented/mixed reality 3D models with near real-time pathology results. Real-time pathology results are generated using a machine learning analysis of biopsy pathology images in conjunction with a predictive model based on the patient's clinical and demographic data and radiographic imaging data. The apparatuses and methods can be used to provide more accurate intraoperative visualization, tracking, and determination of tumor margins to improve resection extent and any intraoperative or postoperative adjuvant therapy.
Core Innovation
A method and an apparatus provide intraoperative surgery guidance by generating, with a processor, a 3D model of an area of interest on a patient's body and displaying a representation of the area of interest as an overlay over a user's view through a head-mounted display device during a surgical operation. The overlay representation includes pathology characteristics associated with the area of interest, so that pathology information is presented in situ as part of the displayed 3D model.
During the surgical operation, the system detects tissue removal and a section location associated with the tissue removal, and associates the section location with a pathology result associated with the tissue removal in the 3D model. Based on the pathology result, the system updates an indicator associated with the section location in the representation displayed on the head-mounted display device, thereby providing an updated representation of the area of interest during the surgical operation.
The pathology result and updated 3D model are further used to convert and/or embed the pathology result into geo-tagged DICOM image files for postoperative treatment planning and monitoring, and pathology-derived outputs can be used to predict prognostic outcomes. The described architecture supports rapid pathology results, including near real-time pathology results using artificial intelligence, and can update tumor surface or mesh boundaries and color-coded indicators to support determination of tumor margins.
Claims Coverage
The document provides two independent claims (clm-00001 and clm-00016). Each independent claim includes the same core inventive concept: creating a 3D model, displaying it as a head-mounted AR/MR overlay with pathology characteristics, detecting tissue removal with a section location, associating the location with a pathology result in the 3D model, and updating a corresponding in-display indicator during the operation.
Generating and displaying a 3D model overlay including pathology characteristics
Generating, with a processor of a computer system, a 3D model of an area of interest on a patient's body; based on the 3D model, providing a representation displayed on a head-mounted display device as an overlay over the user's view during a surgical operation, the representation including pathology characteristics associated with the area of interest.
Detecting tissue removal section location and associating it with a pathology result in the 3D model
Detecting, with the processor during the surgical operation, a tissue removal and a section location associated with the tissue removal; associating the section location with a pathology result associated with the tissue removal in the 3D model of the area of interest to provide an updated 3D model of the area of interest during the surgical operation.
Updating an indicator in the AR/MR representation based on the pathology result
Based on the pathology result, updating an indicator associated with the section location of the tissue removal in the representation displayed on the head-mounted display device to provide an updated representation of the area of interest during the surgical operation.
Apparatus architecture with central server updating the head-mounted indicator during surgery
A computer system operable to generate a 3D model of an area of interest on a patient's body; a head-mounted augmented/mixed reality display device worn by a user, for generating a representation of the area of interest based on the 3D model displayed as an overlay during a surgical operation, wherein the representation includes pathology characteristics; and a central server operable to communicate during the surgical operation to detect a tissue removal and a section location, associate the section location with a pathology result in the 3D model to provide an updated 3D model during the surgical operation, and based on the pathology result update an indicator associated with the section location in the representation displayed on the head-mounted display device.
Across both independent claims, the central inventive coverage lies in combining a 3D model of the area of interest with an AR/MR head-mounted overlay that includes pathology characteristics, detecting tissue removal and a section location during the operation, associating that location with a pathology result within the 3D model to create an updated model, and updating an indicator in the overlay representation during the surgical operation.
Stated Advantages
Provides an updated AR/MR representation of the area of interest during the surgical operation by updating indicators based on pathology results.
Enables incorporation of pathology characteristics into an overlay representation displayed during surgery.
Supports postoperative treatment planning and monitoring by using geo-tagged DICOM image files containing the pathology result.
Supports rapid pathology results, including near real-time pathology results using artificial intelligence.
Enables updated tumor surface/mesh boundaries and color-coded indicators associated with pathology-derived assessments for tumor margin determination.
Documented Applications
Intraoperative surgery guidance during a surgical operation with a head-mounted augmented/mixed reality overlay including pathology characteristics.
Neurosurgery (cranial neurosurgery is emphasized).
Postoperative treatment planning and monitoring using geo-tagged DICOM image files derived from the updated 3D model and pathology result.
Predicting prognostic outcomes using machine learning based on pathology images and patient clinical/demographic data.
Interested in licensing this patent?