Systems and methods for multi-modal sensing of depth in vision systems for automated surgical robots
Inventors
Calef, Thomas J. • Chen, Tina P. • DeMaio, Emanuel • Chen, Tony • Buharin, Vasiliy Evgenyevich • Ruehlman, Michael G.
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11857153-B2
Publication Date
2024-01-02
Expiration Date
Abstract
Systems and methods for multi-modal sensing of three-dimensional position information of the surface of an object are disclosed. In particular, multiple visualization modalities are each used to collect distinctive positional information of a surface of an object. Each of the computed positional information is combined using weighting factors to compute a final, weighted three-dimensional position. In various embodiments, a first depth may be recorded using fiducial markers, a second depth may be recorded using a structured light pattern, and a third depth may be recorded using a light-field camera. Weighting factors may be applied to each of the recorded depths and a final, weighted depth may be computed.
Core Innovation
The document describes multimodal 3D sensing for an automated surgical robot. An imaging system captures a surgical scene using multiple imaging modalities, including baseline fiducial markers, structured light patterns, and light-field or plenoptic imaging, to obtain multiple images that include markers on or near the object and to compute depth measurements and three-dimensional positional information for at least a portion of the object.
For each imaging modality, the document computes separate depth measurements and three-dimensional positional data, and then fuses the modality-specific depth information into a final depth/three-dimensional coordinate. The fusion is performed using one or more weights assigned to the depth measurements, where each weight has a value between zero and one and the sum of the weights equals one, and the depth measurements are weighted based at least in part on the type of imaging modality used to obtain the images.
The document further describes how the weighting can be based on image/pattern contrast, saturation, marker recognition, and pixel-level confidence, including assigning zero weight to invalid pixels. It also includes fiducial marker types and structured light projection, and depth maps can be post-processed, including filtering, subsampling, hole filling, and extrapolation.
Claims Coverage
Independent claim coverage is provided by three independent claims. Each independent claim includes a multimodal surgical depth-sensing system that computes multiple depth measurements from multiple imaging modalities and determines positional information by assigning normalized modality-dependent weights to the depth measurements.
Multimodal depth measurements with modality-weighted fusion
An imaging system obtains a plurality of images of an object in a surgical scene using a plurality of different imaging modalities comprising at least two of RGB imaging, infrared imaging, depth imaging, fiducial marker imaging, structured light pattern imaging, and light field imaging. The processor computes a plurality of depth measurements for at least a portion of the object and determines positional information based on the depth measurements, wherein one or more weights are assigned to the depth measurements, each weight has a value between zero and one, the sum equals one, and the depth measurements are weighted at least in part based on a type of imaging modality used.
Normalized weights within a predefined range for modality-weighted fusion
An imaging system obtains a plurality of images of an object in a surgical scene using a plurality of different imaging modalities including at least RGB imaging, infrared imaging, depth imaging, fiducial marker imaging, structured light pattern imaging, and light field imaging. The processor computes a plurality of depth measurements and determines positional information based on the depth measurements, wherein one or more normalized weights have values between a predefined range, and the depth measurements are weighted based on the type of imaging modality used to obtain the images.
Weights constrained between first and second values with unity-sum for modality-weighted fusion
An imaging system obtains a plurality of images of an object in a surgical scene using a plurality of different imaging modalities comprising at least two of RGB imaging, infrared imaging, depth imaging, fiducial marker imaging, structured light pattern imaging, and light field imaging. The processor computes a plurality of depth measurements and determines positional information based on the depth measurements, wherein one or more weights each has a value between a first value and a second value greater than the first value, the sum of the weights sums to unity, and the depth measurements are weighted based at least in part on the type of imaging modality used to obtain the images.
Across the independent claims, the core inventive concept is a surgical depth-sensing system that computes multiple depth measurements from multiple imaging modalities and determines positional information using normalized weights assigned to the depth measurements, with the weighting performed at least in part based on the imaging modality type. The independent claims further constrain the weight values via unity-sum and, in variant wording, predefined or bounded ranges.
Stated Advantages
Not explicitly described in patent.
Documented Applications
Not explicitly described in patent.
Interested in licensing this patent?