Methods for optical tracking and surface acquisition in surgical environments and devices thereof
Inventors
Marti, Gaëtan • Hälg, Maurice • SIVAGNANASELVAM, Ranjith Steve
Assignees
Smith and Nephew Orthopaedics AG • Smith and Nephew Asia Pacific Pte Ltd • Smith and Nephew Inc
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12402954-B2
Publication Date
2025-09-02
Expiration Date
Abstract
A computer assisted system is disclosed that includes an optical tracking system and one or more computing devices. The optical tracking system includes an RGB sensor and is configured to capture color images of an environment in the visible light spectrum and tracking images of fiducials in the environment in a near-infrared spectrum. The computer assisted system is configured to generate a color image of the environment using the color images, identify fiducial locations using the tracking images, generate depth maps from the color images, reconstruct three-dimensional surfaces of structures based on the depth maps, and output a display comprising the reconstructed three-dimensional surface and one or more surgical objects that are associated with the tracked fiducials. The computer assisted system can further include a monitor or a head-mounted display (HMD) configured to present augmented reality (AR) images during a procedure.
Core Innovation
The invention provides an optical tracking system with an optical module comprising an image sensor and a structured light module configured to project one or more patterns in the near-infrared (NIR) spectrum into an environment. The system executes programmed instructions on one or more processors to project the patterns and capture first and second images with the optical module. The second image is processed to determine tracking information that comprises at least a three-dimensional position of each of one or more fiducials within the environment.
The system generates at least one depth map from the first image and reconstructs a three-dimensional surface of one or more structures in the environment based on the generated depth map. The reconstructed three-dimensional surface is output together with an indication of one or more additional objects. At least one fiducial is affixed to each additional object, and each additional object is located based on the tracking information.
Further embodiments capture a plurality of first images and a plurality of second images. The system generates depth maps for each of the plurality of first images, reconstructs respective three-dimensional surfaces, and stitches the plurality of three-dimensional surfaces together to generate a complete three-dimensional surface. The system then outputs the complete three-dimensional surface along with the indication of one or more additional objects located based on the tracking information.
Claims Coverage
The document includes three independent claims. Across these claims, there are inventive features related to NIR structured-light projection, fiducial-based 3D tracking from image processing, depth-map generation for 3D surface reconstruction, and outputting the reconstructed surface with additional objects located using tracked fiducials.
NIR structured light for fiducial tracking with depth-map surface reconstruction
An optical tracking system having an optical module with an image sensor and a structured light module projecting one or more NIR patterns into an environment, capturing a first image and a second image, processing the second image to determine tracking information comprising at least a three-dimensional position of fiducials, generating at least one depth map from the first image, reconstructing a three-dimensional surface based on the depth map, and outputting the reconstructed three-dimensional surface with an indication of additional objects located based on tracking information from fiducials affixed to the additional objects.
Stitched multi-surface complete 3D reconstruction with tracked additional objects
An optical tracking system capturing a plurality of first images and a plurality of second images, processing each of the plurality of second images to determine tracking information comprising at least a three-dimensional position of fiducials, generating at least one depth map from each first image and reconstructing a three-dimensional surface for each depth map, stitching the plurality of three-dimensional surfaces to generate a complete three-dimensional surface, and outputting the complete three-dimensional surface with an indication of additional objects located based on tracking information using fiducials affixed to the additional objects.
Non-transitory medium for NIR pattern projection, 3D fiducial tracking, and depth-map reconstruction
A non-transitory computer readable medium whose executable instructions cause processors to project a pattern in the NIR spectrum with a structured light module, capture a first image during projection of the pattern and a second image, process the second image to determine tracking information comprising at least a three-dimensional position of fiducials, generate at least one depth map from the first image, reconstruct a three-dimensional surface based on the depth map, and output the reconstructed three-dimensional surface with an indication of additional objects located based on tracking information where fiducials are affixed to each additional object.
Across the independent claims, the system and medium project NIR structured-light patterns, use image capture to compute 3D fiducial positions, generate depth maps from first images to reconstruct 3D surfaces, and output the reconstructed, stitched where applicable, 3D surface together with locations of additional objects determined from tracked fiducials.
Stated Advantages
Not explicitly described in patent.
Documented Applications
Not explicitly described in patent.
Interested in licensing this patent?