Interactive user interfaces for minimally invasive telesurgical systems

Inventors

DiMaio, Simon P. • Hasser, Christopher J. • Taylor, Russell H. • Larkin, David Q. • Kazanzides, Peter • Deguet, Anton • Vagvolgyi, Balazs Peter • Leven, Joshua

Assignees

Johns Hopkins University • Intuitive Surgical Inc • Intuitive Surgical Operations Inc

Abstract

In one embodiment of the invention, a a minimally invasive surgical system is disclosed. The system configured to capture and display camera images of a surgical site on at least one display device at a surgeon console; switch out of a following mode and into a masters-as-mice (MaM) mode; overlay a graphical user interface (GUI) including an interactive graphical object onto the camera images; and render a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

Core Innovation

The invention describes interactive user interfaces for a minimally invasive telesurgical robotic system that includes a surgical assistant workstation (SAW). The SAW captures stereo endoscopic images and ultrasound images, and presents these medical images through a stereoscopic display while overlaying a graphical user interface (GUI) with interactive graphical objects and tool icons adjacent to surgical instruments.

A 3D pointer is rendered based on a state controlled by master tool manipulators (MTMs). In a masters-as-mice (MaM) mode, MTM motion is repurposed as 3D cursor/pointer control, and the grip provides click and drag interaction; outside of MaM mode, MTM motion is coupled to slave instruments, and the interface supports interactive operation consistent with the selected mode.

The SAW architecture includes video acquisition, calibration/registration, tool tracking, and overlay/rendering blocks, together with data management and interaction components. It further includes a window manager concept, context-sensitive menus, icon-driven actions, an ultrasound "flashlight" image window, inset windows, and tools for interactive manipulation of medical image volumes, including pan/zoom/rotate and slice-plane selection.

The interface also supports mentoring via telestration between a supervisor console and a trainee console. It includes GUI-driven virtual fixture setup, including forbidden regions and guidance constraints, for generating a forbidden region virtual fixture and a guidance virtual fixture, and integrates image/data formats such as DICOM and MRML in the interaction and rendering workflow.

Claims Coverage

Not explicitly described in patent.

Not explicitly described in patent.

Stated Advantages

Documented Applications