System and method for identifying activity in an area using a video camera and an audio sensor
Inventors
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12437539-B2
Publication Date
2025-10-07
Expiration Date
Abstract
Identifying activity in an area even during periods of poor visibility using a video camera and an audio sensor are disclosed. The video camera is used to identify visible events of interest and the audio sensor is used to capture audio occurring temporally with the identified visible events of interest. A sound profile is determined for each of the identified visible events of interest based on sounds captured by the audio sensor during the corresponding identified visible event of interest. Then, during a time of poor visibility, a subsequent sound event is identified in a subsequent audio stream captured by the audio sensor. One or more sound characteristics of the subsequent sound event are compared with the sound profiles associated with each of the identified visible events of interest, and if there is a match, one or more matching sound profiles are filtered out from the subsequent audio stream.
Core Innovation
The invention relates to identifying activity in an area during periods of poor visibility by combining a video camera and an audio sensor. A sound event in an audio stream is identified, and when there is a match to a predetermined sound profile in an audio library associated with one or more events of interest, a matching event of interest is identified. Sound characteristics associated with the matching event of interest are removed, including filtering out one or more spectral components while passing remaining spectral components to create a modified audio stream.
After the sound characteristics associated with the matching event of interest are removed, the modified audio stream is analyzed for an abnormal sound remaining in the modified audio stream. When the abnormal sound is detected, an alert is issued. The method also addresses situations where sounds of interest may be at least partially masked by obstructing sounds associated with obstructing events.
Sounds in the audio stream are compared to one or more predetermined sound profiles associated with obstructing events, a matching obstructing event is identified upon a match, and sound characteristics associated with that matching obstructing event are removed using spectral-component filtering. The modified audio stream is then analyzed for the one or more sounds of interest, and an alert is issued when the sounds of interest are detected.
Claims Coverage
The document provides three independent claims, which collectively cover audio-event characterization matched against predetermined sound profiles, spectral-component filtering to remove sound characteristics tied to matched events, subsequent analysis of the modified audio for abnormal sounds or sounds of interest, and issuing alerts when detections occur. The system claim adds poor-visibility operation by gating the audio-processing workflow based on whether legible video was captured for the identified sound event.
Sound-event matching to predetermined sound profiles in an audio library
Identify a sound event in an audio stream; compare one or more sound characteristics of the sound event with one or more predetermined sound profiles stored in an audio library, wherein each predetermined sound profile is associated with one or more events of interest, and when there is a match, identify a matching event of interest.
Filtering spectral components associated with the matching event of interest to form a modified audio stream
Remove from the audio stream one or more sound characteristics associated with the matching event of interest, including filtering out one or more spectral components from the audio stream that are associated with the matching event of interest while passing the remaining spectral components, resulting in a modified audio stream.
Analyzing modified audio for abnormal sounds and issuing an alert
Analyze the modified audio stream for an abnormal sound remaining in the modified audio stream; issue an alert when the abnormal sound is detected in the modified audio stream.
Revealing sounds of interest masked by obstructing sounds via obstructing-event profile matching
Compare sounds in the audio stream to one or more predetermined sound profiles, wherein each predetermined sound profile is associated with one or more obstructing events, and when there is a match, identify a matching obstructing event; remove from the audio stream one or more sound characteristics associated with the matching obstructing event, including filtering out one or more spectral components associated with the matching obstructing event while passing remaining spectral components, resulting in a modified audio stream; analyze the modified audio stream for the one or more sounds of interest; issue an alert when one or more sounds of interest are detected.
Poor-visibility activity identification system using video camera and audio sensor
Provide a video camera and an audio sensor; with a processor operatively coupled to the video camera and the audio sensor, identify a sound event in an audio stream; determine whether a legible video was captured by the video camera of the identified sound event; when no legible video was captured, compare one or more sound characteristics of the sound event with one or more predetermined sound profiles in an audio library to identify a matching event of interest; remove associated sound characteristics including filtering spectral components to create a modified audio stream; analyze the modified audio stream for an abnormal sound remaining; issue an alert when the abnormal sound is detected in the modified audio stream.
Across the independent claims, the claim coverage centers on audio-event matching to predetermined sound profiles, spectral-component filtering to remove sound characteristics associated with matched events, analysis of the modified audio for abnormal sounds or sounds of interest, and alerting upon detection. The system claim adds poor-visibility operation by using video capture legibility to control the audio-processing workflow.
Stated Advantages
Identifies activity in an area even during periods of poor visibility.
Determines whether one or more abnormal sounds are present in the audio stream after removing sound characteristics associated with a matching event of interest.
Reveals one or more sounds of interest that are at least partially masked by obstructing sounds by filtering out obstructing-event spectral components.
Issues an alert when an abnormal sound or one or more sounds of interest are detected in the modified audio stream.
Documented Applications
Monitoring in urban “hot spots” and “smart city” contexts using an intelligent sound classification workflow that performs spectral filtering and sound classification during poor visibility.
Handling cases where no motion is detected by using weather information to create weather sound profiles for masking/removal during audio analysis.
Detection of sounds of interest such as talking/shouting/chanting/screaming/laughing/sneezing/coughing/footsteps/running footsteps.
Interested in licensing this patent?