System and method for determining depth perception in vivo in a surgical robotic system
Inventors
Santini, Fabrizio • KHALIFA, Sammy
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12106502-B2
Publication Date
2024-10-01
Expiration Date
Abstract
A system and method for generating a depth map from image data in a surgical robotic system that employs a robotic subsystem having a camera assembly with first and second cameras for generating image data. The system and method generates based on the image data a plurality of depth maps, and then converts the plurality of depth maps into a single combined depth map having distance data associated therewith. The system and method can then control the camera assembly based on the distance data in the single combined depth map.
Core Innovation
The invention relates to an in-vivo surgical robotic system that performs depth perception using a camera assembly that includes a first camera and a second camera for generating image data. Based on the image data, a depth perception subsystem generates a plurality of depth maps and converts the plurality of depth maps into a single combined depth map having distance data associated therewith.
Multiple depth cues are produced as separate depth maps, including autofocus-derived depth maps, parallax-derived depth maps, and a disparity-derived depth map. The parallax processing includes segmentation and shifted-image motion measurement with camera position data, and the disparity processing can use a layered R-CNN and disparity-to-depth using camera geometry and epipolar geometry to produce disparity depth map information.
A depth map conversion unit fuses the plurality of depth maps into the single combined depth map. The fusion includes per-point confidence value generation using an R-CNN encoder-decoder architecture with Softmax, and the resulting distance data and confidence values are used to control the camera assembly and robotic arms for safer navigation, collision avoidance, automated traversal, and augmented or virtual overlay of surgical information.
Claims Coverage
The partial content includes two independent claims: a surgical robotic system claim and a method claim. Across these independent claims, there are three main inventive feature groupings: a camera assembly with first and second cameras generating image data, generating multiple depth maps and converting them into a single combined depth map with distance data, and using the distance data to control the camera assembly and the robotic subsystem.
Depth map generation and conversion to a single combined depth map with distance data
A depth perception subsystem receives image data generated by a first camera and a second camera, generates a plurality of depth maps based on the image data, and converts the plurality of depth maps into a single combined depth map having distance data associated therewith.
Camera-controlled depth map generation using a first and a second camera
A method generates image data using a robotic subsystem including a camera assembly comprising a first camera and a second camera, generates a plurality of depth maps based on the image data, converts the plurality of depth maps into a single combined depth map having distance data associated therewith, and controls the camera assembly based on the distance data in the single combined depth map.
Both independent claims center on producing a plurality of depth maps from image data generated by a first and second camera, fusing them into a single combined depth map with distance data, and using that distance data for control of the camera assembly within a surgical robotic system.
Stated Advantages
Safer navigation and collision avoidance.
Automated traversal.
Augmented or virtual overlay of surgical information.
Documented Applications
In-vivo surgical robotics with depth perception for safer navigation, collision avoidance, automated traversal, and augmented or virtual overlay of surgical information.
Interested in licensing this patent?