Radar-based methods and apparatus for communication and interpretation of sign languages
Inventors
Gurbuz, Sevgi Zubeyde • Gurbuz, Ali Cafer • Crawford, Chris • Griffin, Darrin
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11301672-B2
Publication Date
2022-04-12
Expiration Date
Abstract
Disclosed herein are methods, apparatus and computer program product for radar-based communication and interpretation of sign languages such as American Sign language (ASL) comprising detecting, using a radar system comprising a computing device, sign language gestures, wherein said detected sign language gestures comprise radar data; analyzing the radar data using a trained neural network executing on the computing device to determine word or phrases intended by the sign language gestures; and outputting the determined words or phrases in a visible or audible format.
Core Innovation
The invention relates to radar-based communication and interpretation of sign languages by detecting sign language gestures with a radar system comprising a computing device, where the detected gestures comprise radar data. The radar data is analyzed using a trained neural network executing on the computing device to determine word or phrases intended by the sign language gestures, and the determined words or phrases are output in a visible or audible format.
The radar system is characterized as a cognitive dual-mode radar system. The radar-based analysis is structured as a dynamic classification scheme that decomposes radar measurements into gross and fine movements using time-frequency-range analysis, wherein the time-frequency-range analysis comprises cepstral filtering.
The radar data is analyzed across multiple domains including time-frequency, time-range, and 3D representations. The invention also models the time-varying nature of the radar data using recurrent neural networks (RNN), performs recognition of sequences of gestures using Connectionist Temporal Classification (CTC), and enforces certain rules to constrain potential sequences of gestures, enabling identification of transition periods between gestures or sequences of gestures.
Claims Coverage
The document provides three independent claims covering a method, an apparatus, and a non-transitory computer program product. Each independent claim includes shared inventive features directed to radar detection of sign language gestures, neural-network analysis of radar data to determine intended words or phrases, and visible or audible output, with the radar system or transceiver being a cognitive dual-mode radar system in the method and apparatus claims.
Radar-based detection of sign language gestures using radar data
Detecting, using a radar system comprising a computing device, sign language gestures, wherein the detected sign language gestures comprise radar data.
Neural-network analysis to determine intended words or phrases
Analyzing the radar data using a trained neural network executing on the computing device to determine word or phrases intended by the sign language gestures.
Visible or audible output of determined words or phrases
Outputting the determined words or phrases in a visible or audible format.
Cognitive dual-mode radar system
Wherein the radar system comprises a cognitive dual-mode radar system.
Radar transceiver with processor, memory, and instructions
A radar transceiver, a memory, and a processor in communication with the memory and the radar transceiver, wherein the processor executes computer-readable instructions stored in the memory that cause the processor to detect sign language gestures, analyze the radar data using a trained neural network, and output the determined words or phrases in a visible or audible format.
Multi-step trained-neural-network radar analysis pipeline for recognition
Decomposition of the radar data into gross and fine movements using time-frequency-range analysis of radar measurements, wherein the time-frequency-range analysis comprises cepstral filtering; analysis across time-frequency, time-range, and 3D representations; modeling of time-varying radar data using recurrent neural networks (RNN); recognition of sequences of gestures using Connectionist Temporal Classification (CTC); and enforcement of rules that constrain potential sequences of gestures and enable identification of transition periods between gestures or sequences of gestures.
Cognitive dual-mode radar transceiver
Wherein the radar transceiver comprises a cognitive dual-mode radar system.
Across the independent claims, the core coverage centers on detecting sign language gestures using radar data, analyzing that radar data with a trained neural network to determine intended words or phrases, and outputting the result in a visible or audible format, with the radar system or transceiver being a cognitive dual-mode radar system. The program-product claim further includes the time-frequency-range analysis with cepstral filtering, multi-domain representations, RNN modeling, CTC sequence recognition, and rule constraints enabling identification of transition periods.
Stated Advantages
Documented Applications
Not explicitly described in patent.
Interested in licensing this patent?