Single image sensor for capturing mixed structured-light images and regular images
Inventors
Wang, Kang-Huai • Xu, Yi • Wilson, Gordon C. • Wu, Chenyu
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-10742909-B2
Publication Date
2020-08-11
Expiration Date
Abstract
A method and device for improving the accuracy of depth information derived from a structured-light image for a regular image are disclosed. In one example, an additional structured-light image is captured before a first structured-light image or after a regular image. The depth information for the regular image can be derived from the first structured-light image and corrected by incorporating depth information from the additional structured-light image. A model for depth information can be used to predict or interpolate depth information for the regular image. In another example, two regular sub-images may be captured with a structured-light image in between. If substantial frame differences or substantial global motion vector/block motion vectors are detected, the two regular sub-images will not be combined in order to avoid possible motion smear. Instead, one of the two sub-images will be selected and scaled as the output regular image.
Core Innovation
The invention relates to deriving depth information for one regular image from a structured-light image using a camera comprising an image sensor having a pixel array. The camera projects non-structured light to a scene, applies reset signals to reset rows of the pixel array, and exposes the rows to obtain analog signals. It captures a first regular sub-image and then projects structured light to capture the structured-light image between two non-structured-light captures.
The method further captures a second regular sub-image from the same pixel array in a second frame period by applying additional reset signals and exposing the rows to the non-structured light. Disparity is determined between the first regular sub-image and the second regular sub-image, and a regular image is generated by combining the first regular sub-image and the second regular sub-image if the disparity is smaller than a threshold, and otherwise using either the first regular sub-image or the second regular sub-image.
The regular image is provided together with the structured-light image, where the structured-light image is used to derive depth information for the regular image. The disclosed approach further refines how disparity is evaluated using mean squared error or sum of absolute differences, and can incorporate motion information to assist motion-compensated sub-image combination. Additionally, the invention discloses applying non-linear or piece-wise linear pixel value transformation to reduce pixel value range before combining sub-images.
Claims Coverage
The patent provides two independent claims. Across the independent claims, the inventive features include capturing first and second regular sub-images around a structured-light image, using disparity to combine or select the regular sub-images, and using the structured-light image to derive depth information for the regular image.
Deriving depth from structured-light for a regular image using two regular sub-images with disparity-based combine or select
Projecting non-structured light and capturing a first regular sub-image, projecting structured light and capturing a structured-light image between first and second regular sub-images, projecting non-structured light and capturing a second regular sub-image, determining disparity between the first and second regular sub-images, generating a regular image by combining the first and second regular sub-images if disparity is smaller than a threshold and otherwise generating the regular image using either the first or the second regular sub-image, providing the regular image, and providing the structured-light image to derive depth information for the regular image.
Endoscope system with structured and non-structured light, disparity-based regular image generation, and depth derivation
An endoscope system comprising an image sensor and a structured light source and a non-structured light source, timing and control circuits arranged to project non-structured light and capture a first regular sub-image, project structured light and capture a structured-light image between non-structured-light captures, project non-structured light and capture a second regular sub-image, determine disparity between the first and second regular sub-images, generate a regular image by combining the first and second regular sub-images if disparity is smaller than a threshold and otherwise using either the first or the second regular sub-image, provide the regular image, and provide the structured-light image to derive depth information for the regular image.
Both independent claims center on deriving depth information by using a structured-light image in combination with a regular image assembled from first and second regular sub-images. The regular image generation is controlled by disparity between the two regular sub-images, using a threshold to decide between combining or selecting one sub-image.
Stated Advantages
Documented Applications
No documented applications found
Interested in licensing this patent?