Spherical image based registration and self-localization for onsite and offsite viewing
Inventors
Boyless, Nathaniel • Liu, Jiayi • Siva, Sriram • Zhang, Hao
Assignees
Metcalf Archaeological Consultants Inc • Colorado School of Mines
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
Systems and methods for image registration and self-localization for onsite and offsite viewing are provided. In one aspect, systems and methods for spherical image based registration and self-localization for onsite and offsite viewing, such as augmented reality viewing and virtual reality viewing, are provided. In one embodiment, a system includes a portable electric device fitted with a device camera, an external omnidirectional spherical camera, and a remote server. In one embodiment, a method of use employs a set of fiducial markers inserted in images collected by a spherical camera to perform image registration of spherical camera images and device camera images.
Core Innovation
The disclosure provides a system and method for spherical image based self-localization in which a device displays at least one fiduciary marker and positions the device display within the field of view of a spherical camera. The spherical camera collects spherical camera images including the at least one fiduciary marker on the device display, and transmits the spherical camera image data for processing. A transformation of the device display with respect to the spherical camera is determined using at least the spherical camera images, and the transformation is used to determine an absolute device state comprising absolute device orientation and absolute device position.
The disclosed self-localization combines the determined transformation with absolute device state computation. The system includes a device camera, a device communications module, a device processor, a device display, and a device orientation sensor, and a spherical camera with a spherical camera field of view, spherical camera communications module, and spherical camera image collection capability. Dependent refinements include determining the transformation using a weighted matching approach and using external state inputs such as GNSS-measured device state data and device orientation sensor measurements.
In further described embodiments, fiduciary recognition is performed with respect to the spherical camera images and estimation of a pose of the at least one fiduciary marker is performed with respect to the spherical camera images. The disclosure also describes use of a synthetic rectangular fiduciary marker with a wide black border and an inner binary matrix containing encoded information. The overall arrangement supports onsite augmented reality and offsite virtual reality viewing, including contextual data overlay rendering in an augmented reality view based on the determined device state.
Claims Coverage
The partial content provides three independent claims. Across these claims, the inventive features focus on displaying fiduciary markers within a spherical camera field of view, determining a transformation of the device display with respect to the spherical camera from spherical camera images, and using that transformation to determine an absolute device state comprising absolute device orientation and absolute device position, with optional GNSS and sensor-based refinements and fiduciary recognition/pose estimation.
Fiduciary marker displayed within the spherical camera field of view
Displaying at least one fiduciary marker on the device display and positioning the device display within the spherical camera field of view so that the spherical camera images comprise the at least one fiduciary marker on the device display.
Transformation determination of the device display with respect to the spherical camera
Determining a transformation of the device display with respect to the spherical camera using at least the spherical camera images.
Absolute device state computation from the transformation
Determining an absolute device state using the transformation, wherein the absolute device state comprises absolute device orientation and absolute device position, and thereby determining spherical image based self-localization.
Using weighted matching for transformation
Determining the transformation of the device display using at least the spherical camera images comprising the at least one fiduciary marker, wherein the transformation is determined using a weighted matching approach.
Using GNSS-measured device state data and device orientation sensor data
Determining the transformation using the spherical camera images, the GNSS-measured device state data comprising device orientation and device position, and the device sensor orientation data from the device orientation sensor.
Fiduciary recognition and pose estimation of the fiduciary marker
Performing fiduciary recognition with respect to the spherical camera images and performing estimation of a pose of the at least one fiduciary marker with respect to the spherical camera images.
Synthetic rectangular fiduciary marker with inner binary matrix
The at least one fiduciary marker comprises a synthetic rectangular marker of a wide black border and an inner binary matrix that contains encoded information.
Contextual data overlay rendering in augmented reality
Rendering a contextual data overlay on a device display in an augmented reality view.
Across the independent claims, the core claim coverage is the combination of displaying fiduciary markers within a spherical camera field of view, determining a transformation of the device display with respect to the spherical camera from spherical camera images, and computing an absolute device state including absolute device orientation and absolute device position to determine spherical image based self-localization. Additional inventive features include weighted matching for transformation, fusing GNSS-measured device state data and device orientation sensor data, performing fiduciary recognition and pose estimation, using a synthetic rectangular marker structure, and rendering contextual data overlays in augmented reality.
Stated Advantages
Not explicitly described in patent.
Documented Applications
Not explicitly described in patent.
Interested in licensing this patent?