Augmented reality using eye tracking in a robot assisted surgical system
Inventors
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12042240-B2
Publication Date
2024-07-23
Expiration Date
Abstract
A system augments a surgical procedure by providing information to the user about regions of a surgical site the user is viewing on a displayed endoscopic image. The system uses an eye tracker to determine the region of the displayed image being viewed by the user, and applies computer vision to the image data to identify structures, conditions or disease states within the region. The system provides information to the user, generates alerts, or modifies operational settings for surgical devices based on the identified structures, conditions or disease states.
Core Innovation
The invention relates to an augmented reality robotic surgical system that includes a camera positionable in a body cavity, at least one surgical instrument having a distal end positionable within the body cavity, a display, and an eye tracker. The system displays in real time an image of the body cavity captured by the camera, and uses input from the eye tracker to determine the region of the displayed image being viewed by the user. It performs, in real time, computer vision analysis of an area of the image corresponding to the viewed region.
Based on the computer vision analysis, the system identifies blood vessels, organs, tissue types, tumors, structures to be avoided by the surgical instrument, or regions to be avoided by the surgical instrument within the region from image data obtained by the camera. The system obtains information from a database regarding the identified items within the body cavity, using the identified blood vessels, organs, tissue types, tumors, structures to be avoided, or regions to be avoided as retrieval subjects.
The system generates an overlay to alert the user to the identified blood vessels, organs, tissue types, tumors, or structures to be avoided, or regions to be avoided, and displays the overlay in real time on the display overlaying the captured image. The provided summary further describes augmented outputs such as overlays, graphics, icons, text, and auditory alerts.
Claims Coverage
The document provides two independent claims. Across these claims, four core inventive elements recur: eye-tracker-based determination of the viewed region on an endoscopic display, real-time computer vision analysis restricted to that viewed region for identifying anatomical and avoidance targets, database-informed generation of a real-time overlay alerting the user, and real-time display of the body cavity image.
Eye-tracker-based viewed region determination
Based on input from the eye tracker, determine the region of the displayed image being viewed by the user.
Real-time region-limited computer vision identification
Perform, in real time, computer vision analysis of an area of the image determined to correspond to the region being viewed by the user on the display and to, based on the computer vision analysis, identify blood vessels, organs, tissue types, tumors, structures to be avoided by the surgical instrument, or regions to be avoided by the surgical instrument, within the region from image data obtained by the camera.
Database information retrieval for identified targets
Obtain information from a database regarding the identified blood vessels, organs, tissue types, tumors, structures to be avoided by the surgical instrument, or regions to be avoided by the surgical instrument, within the body cavity.
Real-time overlay alerting
Generate an overlay to alert the user to the identified blood vessels, organs, tissue types, tumors, or structures to be avoided by the surgical instrument, or regions to be avoided by the surgical instrument, and displaying the overlay in real time on the display overlaying the image.
Real-time display of body cavity image
Cause an image of the body cavity captured by the camera to be displayed in real time on the display.
Claim coverage centers on a system that uses an eye tracker to determine a viewed region on a real-time endoscopic display, performs real-time computer vision analysis of that region to identify blood vessels, organs, tissue types, tumors, and avoidance targets, retrieves corresponding information from a database, and generates a real-time overlay overlaying the image to alert the user.
Stated Advantages
Alerts the user to identified blood vessels, organs, tissue types, tumors, and structures or regions to be avoided by the surgical instrument.
Displays the overlay in real time on the display overlaying the image.
Documented Applications
Augmented reality robotic surgical visualization to identify anatomical structures and provide real-time overlay alerts during endoscopic procedures.
Interested in licensing this patent?