Sensor fusion to validate sound-producing behaviors
Inventors
CHAN, Eduardo • MAULT, James • Wang, David • Leung, Henry • Jones, Sam • Ross, Mark
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11172909-B2
Publication Date
2021-11-16
Expiration Date
Abstract
A method to measure sound-producing behaviors of a subject with a power- and bandwidth-limited electronic device that includes a processor includes measuring, by a microphone communicatively coupled to the processor, sound in a vicinity of the subject to generate an audio data signal that represents the sound. The method also includes measuring, by at least one second sensor communicatively coupled to the processor, at least one parameter other than sound to generate at least a second data signal that represents the at least one parameter other than sound. The method also includes detecting one or more sound-producing behaviors of the subject based on: both the audio data signal and the second data signal; or information derived from both the audio data signal and the second data signal.
Core Innovation
The invention relates to a power- and bandwidth-limited electronic device that measures sound-producing behaviors of a subject using sensor fusion. The device includes a processor, a microphone that measures sound in a vicinity of the subject to generate an audio data signal, and at least one additional sensor that measures at least one parameter other than sound to generate a second data signal. The detection uses both the audio data signal and the second data signal, or information derived from both, to improve sound-producing behavior detection in noisy environments.
The method extracts a plurality of audio features from the audio data signal and a plurality of second features from the second data signal, and classifies each audio feature of a first subset and each second feature of a first subset as indicative of a sound-producing behavior. In a subsequent period of time, the method measures sound again to generate a subsequent audio data signal and extracts a subsequent plurality of audio features. The subsequent audio features are classified as indicative of sound-producing behaviors based on the classification results.
To manage limited device resources, the method determines a quality score of the subsequent audio data signal and uses threshold quality score logic to decide whether to detect or discard. If the quality score exceeds a first threshold quality score, the method detects one or more sound-producing behaviors occurring within the subsequent period of time based exclusively on the first subset of the subsequent plurality of audio features. If the quality score is less than a second threshold quality score lower than the first threshold quality score, the method discards one or both of the subsequent audio data signal and the first subset of the subsequent plurality of audio features without detecting any sound-producing behaviors that occur within the subsequent period of time based thereon.
The system can include wake-up event detection based on the second data signal and can use remote processing by extracting features and behaviors for learning and updating algorithms. A remote server can receive selected extracted features and/or detected sound-producing behaviors with occurrence times while not reporting any of the audio data signal or any of the second data signal. The system can also receive subject-provided annotation inputs that confirm sound-producing behaviors, report those annotations to the remote server to update a classification algorithm, and then receive the updated algorithm from the remote server.
Claims Coverage
The partial content identifies one independent claim with dependent refinements in the provided claim set, yielding inventive features centered on power-/bandwidth-limited operation, wake-up from non-audio sensing, fused audio/non-audio classification, quality-score-based discard/no-detection, and remote feature/behavior reporting and annotation-driven learning.
Power- and bandwidth-limited multi-sensor sound-behavior measurement
A method to measure sound-producing behaviors of a subject with a power- and bandwidth-limited electronic device that includes a processor, the method comprising measuring a second data signal representing at least one parameter other than sound, providing power to a microphone to measure sound and generate an audio data signal, and detecting one or more sound-producing behaviors based on both the audio data signal and the second data signal or information derived from both.
Wake-up event from non-sound parameter signal
Detecting a wake-up event from the second data signal.
Feature extraction and classification using audio features and second features
Extracting a plurality of audio features from the audio data signal and a plurality of second features from the second data signal, classifying each audio feature of a first subset and each feature of a first subset of the plurality of second features as indicative of a sound-producing behavior, and detecting the one or more sound-producing behaviors based on the classifying.
Subsequent-period detection with quality-score thresholding and discard without detection
Measuring sound in the vicinity of the subject within a subsequent period of time to generate a subsequent audio data signal, extracting a subsequent plurality of audio features, classifying each audio feature of a first subset as indicative of a sound-producing behavior, determining a quality score of the subsequent audio data signal, and then either detecting based exclusively on the first subset when the quality score exceeds a first threshold quality score, or discarding one or both of the subsequent audio data signal and the first subset of the subsequent plurality of audio features without detecting any sound-producing behaviors when the quality score is less than a second threshold quality score lower than the first threshold quality score.
Subclassification of sound-producing behavior types from fused features
Detecting one or more sound-producing behaviors by subclassifying each audio feature of a first subset of the plurality of audio features and each feature of a first subset of the plurality of second features to determine corresponding types of sound-producing behaviors.
Enumerated sound-producing behavior types
The sound-producing behaviors include at least one of sneezing, wheezing, shortness of breath, chewing, swallowing, masturbation, sex, coughing, vomiting, and shouting.
Remote reporting of selected features/behaviors without raw audio or second signals
Reporting to a remote server selected extracted features and/or detected sound-producing behaviors with occurrence times, without reporting any of the audio data signal or any of the second data signal.
Subject-provided annotation inputs to update classification algorithm
Receiving subject-provided annotation inputs that confirm sound-producing behaviors, reporting those annotations to a remote server to update an algorithm for classifying, and receiving the updated algorithm from the remote server.
Across the independent claim and dependent refinements, the claim coverage centers on detecting sound-producing behaviors with a microphone and at least one non-audio sensor using extracted audio features and extracted second features, initiating detection via a wake-up event from the second data signal, and performing subsequent-period classification using quality-score thresholding with an explicit discard/no-detection path. Optional dependent features further specify behavior type subclassification and examples, and add remote reporting and an annotation-driven algorithm update loop.
Stated Advantages
Improves sound-producing behavior detection by using both audio data signal and second data signal to address noisy environments.
Reduces bandwidth/power usage by being a power- and bandwidth-limited electronic device and by discarding low-quality data without detecting sound-producing behaviors.
Enables remote learning/updates while not reporting raw audio data signal or second data signal by reporting selected extracted features and/or detected behaviors with occurrence times.
Allows algorithm improvement using subject-provided annotation inputs confirming sound-producing behaviors.
Documented Applications
Detecting sound-producing behaviors of a subject, including at least sneezing, coughing, vomiting, shouting, wheezing, shortness of breath, chewing, swallowing, masturbation, and sex [as enumerated].
Remote server reporting for detection/learning, including sending selected extracted features and/or detected sound-producing behaviors with occurrence times while not reporting raw audio or second signals.
Using subject-provided annotation inputs that confirm sound-producing behaviors to update and retrieve an updated classification algorithm from a remote server.
Interested in licensing this patent?