Knowledge-driven scene priors for semantic audio-visual embodied navigation
Inventors
Francis, Jonathan • Bondi, Luca • Tatiya, Gyan • Navarro, Ingrid
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Assignees
Member
Carnegie Mellon UniversityCarnegie Mellon University is a global research institution based in Pittsburgh, Pennsylvania, recognized for interdisciplinary education, research, and innovation in science, engineering, arts, technology, and social sciences. The university leads advancements in artificial intelligence, robotics, digital health, and performing arts. Located in a technology-driven and culturally rich city, CMU powers real-world impact through research centers, industry engagement, workforce training, and initiatives that shape regional and global communities.
Carnegie Mellon University is a global research institution based in Pittsburgh, Pennsylvania, recognized for interdisciplinary education, research, and innovation in science, engineering, arts, technology, and social sciences. The university leads advancements in artificial intelligence, robotics, digital health, and performing arts. Located in a technology-driven and culturally rich city, CMU powers real-world impact through research centers, industry engagement, workforce training, and initiatives that shape regional and global communities.
Publication Number
US-12586393-B2
Publication Date
2026-03-24
Expiration Date
Abstract
A method of controlling navigation of a device in an environment using machine learning (ML) models includes receiving visual and audio observation data of the environment as sensed by the device, determining classification scores for objects and regions in the environment based on the visual and audio observation data, encoding visual information based on the classification scores, determining audio-semantic feature embeddings based at least in part on the classification scores, the audio-semantic feature embeddings indicating spatial relationships between objects in the environment, between regions in the environment, and between objects and regions in the environment, and determining and outputting, based on the encoded visual information and the audio-semantic feature embeddings, a state representation corresponding to a state of the device within the environment.
Core Innovation
A method of controlling navigation of a device in an environment using machine learning models receives visual and audio observation data of the environment as sensed by the device, and determines classification scores for objects and regions based on the visual and audio observation data. The method encodes visual information based on the classification scores and determines a distance and direction of a sounding object from the device based on the audio observation data and a direct-to-reverberant ratio of an impulse sounding response between the sounding object and the device.
The method determines audio-semantic feature embeddings based at least in part on the classification scores, the determined distance and direction of the sounding object from the device, and the direct-to-reverberant ratio. The audio-semantic feature embeddings indicate spatial relationships between objects in the environment, between regions in the environment, and between objects and regions in the environment. The method determines and outputs, based on the encoded visual information and the audio-semantic feature embeddings, a state representation corresponding to a state of the device within the environment.
The method controls operation of the device based on the state representation. A system and computing device embodiments configure sensors, vision and audio networks, a location predictor, and a policy network to carry out the same relationships between classification scores, audio-derived distance and direction and direct-to-reverberant ratio, audio-semantic feature embeddings, and a state representation for device control. Dependent embodiments further add graph encoder networks and knowledge graph structure for computing audio-semantic and visual-semantic feature embeddings, and specify training-data combinations spanning previously seen and unseen indoor environments and previously heard and unheard sounds.
Claims Coverage
The independent claims cover controlling navigation by fusing visual classification scores with audio-derived spatial cues to produce embeddings that indicate spatial relationships, and then using those embeddings to form a device state representation for controlling the device operation. Three independent-claim families are described, with dependent claims adding graph-encoder and knowledge-graph embedding computation and training-data constraints.
Controlling navigation using classification scores and audio-derived spatial cues
A method of controlling navigation of a device in an environment using machine learning models includes receiving visual and audio observation data, determining classification scores for objects and regions based on the visual and audio observation data, encoding visual information based on the classification scores, determining a distance and direction of a sounding object and a direct-to-reverberant ratio of an impulse sounding response, determining audio-semantic feature embeddings indicating spatial relationships, determining and outputting a state representation based on the encoded visual information and the audio-semantic feature embeddings, and controlling operation of the device based on the state representation.
Navigation control system with vision, audio, location prediction, embeddings, and a policy
A system for controlling navigation of a device in an environment includes sensors configured to receive visual and audio observation data, a vision network configured to determine visual classification scores, a location predictor configured to determine distance and direction of a sounding object and a direct-to-reverberant ratio of an impulse sounding response, an audio network configured to determine audio classification scores and audio-semantic feature embeddings based on the visual classification scores and the determined distance, direction, and direct-to-reverberant ratio, and a policy network configured to encode visual information and determine and output a state representation based on the encoded visual information and the audio-semantic feature embeddings, where one or more processing devices control operation of the device based on the state representation.
Computing device producing a spatial state representation from multimodal embeddings for navigation control
A computing device configured to control navigation of a device in an environment using machine learning models is configured to receive visual and audio observation data, determine classification scores for objects and regions based on the visual and audio observation data, encode visual information based on the classification scores, determine distance and direction of a sounding object and a direct-to-reverberant ratio of an impulse sounding response, determine audio-semantic feature embeddings indicating spatial relationships using the classification scores and the determined distance, direction, and direct-to-reverberant ratio, determine and output a state representation based on the encoded visual information and the audio-semantic feature embeddings, and control operation of the device based on the state representation.
Across the independent claims, navigation control is based on combining classification scores for objects and regions with audio-derived distance, direction, and direct-to-reverberant ratio to compute audio-semantic feature embeddings that indicate spatial relationships, and then using those embeddings together with encoded visual information to produce a state representation that drives device control. Dependent claims further specify graph encoder and knowledge graph structures for audio-semantic and visual-semantic embeddings and constrain training via previously seen and unseen indoor environments and previously heard and unheard sounds.
Stated Advantages
Documented Applications
No documented applications found
Interested in licensing this patent?