Augmented reality system and methods for stereoscopic projection and cross-referencing of live x-ray fluoroscopic and computed tomographic c-arm imaging during surgery
Inventors
Yanof, Jeffrey H. • Braido, Peter Nicholas
Assignees
Publication Number
US-12229900-B2
Publication Date
2025-02-18
Expiration Date
2042-08-11
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
A method for performing a procedure on a patient includes acquiring a three-dimensional image of a location of interest on the patient and a two-dimensional image of the location of interest can be acquired. A computer system can relate the three-dimensional image with the two-dimensional image to form a holographic image dataset. The computer system can register the holographic image dataset with the patient. The augmented reality system can render a hologram based on the holographic image dataset from the patient. The hologram can include a projection of the three-dimensional image and a projection of the two-dimensional image. The practitioner can view the hologram with the augmented reality system and perform the procedure on the patient. The practitioner can employ the augmented reality system to visualize a point on the projection of the three-dimensional image and a corresponding point on the projection of the two-dimensional image during the procedure.
Core Innovation
The invention relates to an augmented reality system for use during surgical procedures, which stereoscopically projects, registers, and cross-references real-time live fluoroscopic projection imaging and cone-beam computed tomography (CBCT) imaging results acquired during the procedure. The system uses perspective projection to augment virtual objects derived from imaging systems so that their spatial relationships are congruent with the physical C-arm geometry and pose, thereby providing 3D guidance and navigation relative to the patient and the C-arm system.
The problem being solved is the limitation of current methods that register CT imaging including CBCT with fluoroscopy but display results only on two-dimensional monitors. Such 2D fusion may obscure live fluoroscopy images, limiting spatial understanding. There remains a need for a system that can project multimodal imaging results congruently with the physical imaging devices in three dimensions to improve procedural visualization, accuracy, and efficiency.
The disclosed system acquires a three-dimensional image dataset of a location of interest on the patient (such as CBCT or multidetector row CT) and a two-dimensional image (such as live fluoroscopy). A computer system relates these images geometrically to form a holographic image dataset registered with the patient. An augmented reality system renders a hologram including projections of both the 3D and 2D images for stereoscopic viewing by a practitioner. This allows visualization and cross-referencing of corresponding points on both images during the procedure, enhancing depth perception and spatial understanding for improved surgical guidance.
Claims Coverage
The patent includes two independent claims covering a method and a system for performing a procedure on a patient utilizing an augmented reality system with multimodal imaging integration and holographic visualization.
Integration and relating of 3D and 2D images
Acquiring a three-dimensional image dataset forming a 3D image of a location of interest and acquiring a two-dimensional image of the same location, and relating these images by a computer system to form a three-dimensional holographic image dataset.
Holographic projection and rendering
Projecting the holographic image dataset with the patient and rendering a hologram based on the holographic image dataset including projections of both the 3D and 2D images for stereoscopic viewing by a practitioner.
Cross-referencing of corresponding points on 3D and 2D projections
Permitting the practitioner to visualize and cross-reference a selected point on the projection of the 3D image and a corresponding point on the projection of the 2D image during the procedure, including optionally rendering a line segment between the points.
Use of C-arm system geometry for spatial registration and projection
Employing the geometry and physical pose of a C-arm system in registering and projecting the holograms, including perspective projection parameters such as source-to-detector distance, isocenter, and patient table pose.
Use of head-mounted augmented reality display
Rendering the hologram using an augmented reality system including a head-mounted stereoscopic display wearable by the practitioner to enhance depth perception and spatial understanding.
Tracking of surgical instruments
Integrating and calibrating tracked surgical instruments with the system and showing their real-time movement within the holographic visualization during the procedure.
Interactive manipulation of hologram
Allowing the practitioner to manipulate position, orientation, and scale of the hologram and maintaining stereo projection and cross-referencing relationships during such manipulation.
The claims cover a novel augmented reality system and method combining stereoscopic projection, spatial registration using C-arm geometry, live multimodal medical imaging (3D and 2D), interactive holographic visualization, and tracked instrument integration to improve surgical procedure guidance.
Stated Advantages
Improves the effectiveness, safety, and efficiency of 3D guidance and navigation during interventional procedures.
Provides synergistic advantages of combining 3D CBCT data with live 2D fluoroscopy imaging in real time.
Enhances depth perception and spatial understanding via stereoscopic holographic projection using a head-mounted display.
Facilitates cross-referencing of corresponding points on multimodal images to improve targeting accuracy.
Allows real-time tracking and visualization of surgical instruments within the holographic environment.
Reduces ionizing radiation burden and skin dose by enabling better planning and positioning of physical C-arm based on holographic projections.
Documented Applications
Use during surgical procedures including neurological, cardiac, oncology, orthopedic, gastroenterology, orthodontic, and dental procedures.
Interventional radiology or angiography procedures requiring 3D guidance and live fluoroscopy imaging.
Use with C-arm systems to stereoscopically project and cross-reference cone-beam CT and live fluoroscopic imaging.
Tracking and guidance of interventional devices such as needles and imaging probes during procedures.
Matching respiratory phases of patient ventilation during procedures to improve targeting accuracy.
Interested in licensing this patent?