Mixed reality combination system

Inventors

Cagan, Jonathan • Leduc, Philip • Kabuye, Ernest

Abstract

Disclosed herein is a system providing a mixed reality combination system that pairs augmented reality technology and an inertial measurement unit sensor with 3D printed objects such that user motions tracked by the inertial measurement unit as the user interacts with the 3D printed object is reflected in a virtual environment display of dynamic 3D imagery and augmented reality imagery.

Core Innovation

The invention is a Mixed Reality Combination System that combines an augmented reality environment, a printed environment with one or more 3D printed objects, and a tracking environment for visually tracking a position and orientation of a tool within the target environment. The system includes a user interaction module specifying a user task and including visual aids for assisting a user in completion of the specified user task. The augmented reality environment provides dynamic augmented reality imagery aligned with and projected on the one or more 3D printed objects in a user field-of-view of the target environment in response to motion of a user as tracked by the tracking environment.

The dynamic imagery shows movement of objects in the augmented reality environment in response to user motions provided by the tracking environment. The system predicts a future state of a target object based on the task specified by the user interaction module, manipulates the 3D model representing the target object to show the predicted state, and projects imagery showing the predicted state onto the printed environment as the visual aid. In response to the predicted state, the system uses motion of the user and tracking to update what is displayed in the user field-of-view.

The system additionally generates dynamic 3D imagery, including producing dynamic 3D imagery from a series of 3D images created by stitching multiple 2D images representing successive object states during an object transition, and optionally applying finite element modeling. The claim set also describes examples involving a 3D printed bio-specimen made of collagen and movement and state transitions, with interaction visualization aligned to the printed environment and corresponding predicted state imagery.

Claims Coverage

The patent includes two independent claims. Across these claims, the main inventive features are a mixed reality combination system that aligns dynamic augmented reality imagery to one or more 3D printed objects based on tracked user/tool motion and task-specified prediction of a future target-object state, and a head-mounted display system with spatial mapping and inertial measurement from one or more instrumented tools to provide motion-responsive dynamic 3D imagery projected onto a 3D printed object.

Overall, the independent claims cover aligning motion-responsive dynamic imagery with 3D printed objects, tracking user/tool motion, and using a user-specified task to predict and display a future state by manipulating a target-object 3D model and projecting predicted-state imagery onto the printed environment; a second independent claim further emphasizes spatial mapping in a head-mounted display together with inertial measurement from instrumented tools to drive the dynamic projection.

Stated Advantages

Documented Applications