Identifying convergence of sensor data from first and second sensors within an augmented reality wearable device
Inventors
Lacey, Paul • Miller, Samuel A. • Kramer, Nicholas Atkinson • Lundmark, David Charles
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12444146-B2
Publication Date
2025-10-14
Expiration Date
Abstract
Examples of wearable systems and methods can use multiple inputs (e.g., gesture, head pose, eye gaze, voice, totem, and/or environmental factors (e.g., location)) to determine a command that should be executed and objects in the three-dimensional (3D) environment that should be operated on. The wearable system can detect when different inputs converge together, such as when a user seeks to select a virtual object using multiple inputs such as eye gaze, head pose, hand gesture, and totem input. Upon detecting an input convergence, the wearable system can perform a transmodal filtering scheme that leverages the converged inputs to assist in properly interpreting what command the user is providing or what object the user is targeting.
Core Innovation
The invention describes an augmented reality and mixed reality wearable system that performs transmodal input fusion. The system accesses sensor data from a plurality of sensors of different modalities and identifies convergence events between sensor data from a first sensor and sensor data from a second sensor. During convergence events, the system uses that agreement to condition how sensor data is processed.
A central feature is selectively applying a noise filter to the sensor data from the first sensor during convergence events. The system detects convergence of the sensor data from the first and second sensors, applies the noise filter to the first sensor data, and then detects divergence of the first sensor data from the second sensor data. Based on the detected divergence, the system disables application of the noise filter to the first sensor data.
The disclosed framework is tied to multimodal interaction in a 3D environment, including eye gaze, head pose, hand gesture, voice commands, totem input, environmental factors, EMG, and a camera-based hand gesture sensor. Upon convergence, sensor processing is conditioned to support interpretation of user intent, including target virtual object identification, selection, and command execution in a 3D environment.
Claims Coverage
Two independent claims are explicitly provided: one method claim and one wearable system claim. Across these claims, the inventive core is detecting convergence and divergence between sensor modalities and selectively applying a noise filter to one sensor based on that convergence state.
Detecting convergence events across sensor modalities
Accessing sensor data from a plurality of sensors of different modalities and identifying convergence events of sensor data from a first sensor and sensor data from a second sensor.
Selectively applying noise filter during convergence
During the convergence events, selectively applying a noise filter to the sensor data from the first sensor.
Disabling noise filter based on divergence
Detecting divergence of the sensor data of the first sensor from the sensor data of the second sensor and, based on the divergence, disabling application of the filter to the sensor data from the first sensor.
Wearable system with processor for convergence-conditioned noise filtering
A wearable system comprising a plurality of sensors of different modalities and a hardware processor programmed to access sensor data, identify convergence events between first and second sensor data, and during the convergence events selectively apply a noise filter to the sensor data from the first sensor, including disabling the filter based on divergence.
The independent claims cover both a method and a wearable system in which convergence events across different sensor modalities are detected, a noise filter is selectively applied to a first sensor during convergence, and the filter is disabled when divergence is detected.
Stated Advantages
Reduce uncertainty by applying a noise filter to the first sensor data only during convergence events while limiting latency.
Reduced precision requirements for 3D interaction.
Improved robustness in 3D interaction.
Lower cost via multi-input compensation.
Documented Applications
Using the sensor convergence/divergence among multiple sensors for AR/MR wearable interaction to identify a target virtual object in a three-dimensional environment.
Using convergence between EMG sensor data and a camera-based hand gesture sensor to detect nonverbal symbols for multimodal input fusion in wearable interaction.
Target object selection and command execution in a 3D environment using sensor data from multiple modalities.
Interested in licensing this patent?