Machine learning model capability assessment
Inventors
Lawson, Aaron • McLaren, Mitchell L • Castan Lavilla, Diego
Assignees
Publication Number
US-12035106-B2
Publication Date
2024-07-09
Expiration Date
2041-10-22
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
In some examples, a system includes a storage device; and processing circuitry having access to the storage device. The processing circuitry is configured to receive information indicative of a media dataset, where the media dataset corresponds to an object; and analyze the media dataset to compute a corresponding set of operating condition weight values. Additionally, the processing circuitry is configured to compare the set of operating condition weight values corresponding to the media dataset with a plurality of sets of reference operating condition weight values that each correspond to a different reference media dataset of a plurality of reference media datasets; and determine, based on the comparison of the set of operating condition weight values with the plurality of sets of reference operating condition weight values, an indication of a capability of a trained machine learning model to correctly verify the object in the media dataset.
Core Innovation
The invention provides a system and methods for assessing the capability of a machine learning model to verify objects in a media dataset under varying operating conditions. The processing circuitry receives a media dataset, which may comprise audio or image data corresponding to an object, and analyzes this dataset to compute a set of operating condition weight values—each corresponding to an operating condition such as environmental or behavioral factors. These weight values quantify the presence of specific operating conditions in the dataset.
The system then compares the computed set of operating condition weight values with multiple sets of reference operating condition weight values, each tied to a different reference media dataset. By assessing the similarities between the conditions of the new media dataset and those present in the pool of reference datasets, the system determines an indication of the machine learning model’s capability to correctly verify the object in question. The system can perform further actions based on this assessment, such as calibrating the model, adjusting system parameters, or suggesting the addition of more reference datasets if the current pool is insufficient.
The invention addresses the problem that machine learning models may not accurately determine whether a dataset includes certain characteristics when training data is insufficient, particularly if there is inadequate representation of different operating conditions. By evaluating whether enough similar reference conditions exist for robust calibration or verification, the system ensures reliable performance even when encountering diverse or previously unrepresented operating conditions.
Claims Coverage
There is one independent claim describing the main inventive features of the computing system and method for machine learning model capability assessment.
Computing system for assessing machine learning model capability under varying operating conditions
The system comprises processing circuitry and a storage device. The processing circuitry: - Receives information indicative of a media dataset (audio and/or image data) corresponding to an object. - Analyzes the media dataset to compute a corresponding set of operating condition weight values, with each weight value corresponding to a different operating condition. - Compares the set of operating condition weight values with multiple sets of reference operating condition weight values, each set corresponding to a different reference media dataset. - Determines, based on the comparisons, an indication of the capability of a trained machine learning model to correctly verify the object in the media dataset. - Performs an action based on the determined indication.
Method for analyzing and assessing model capability
- Receiving a media dataset comprising at least one of audio data and image data associated with an object. - Analyzing the dataset to compute a set of operating condition weight values representing various operating conditions. - Comparing the weight values of the media dataset against a plurality of sets of reference operating condition weight values from reference datasets. - Determining, based on the comparisons, an indication of the model's capability to verify the object. - Performing an action based on this assessment.
Non-transitory computer-readable medium with instructions for capability assessment
Instructions, when executed by a processor, cause the system to: - Receive a media dataset (audio/image data) corresponding to an object. - Analyze the media dataset to compute operating condition weight values (each for a different operating condition). - Compare these with several sets of reference operating condition weight values from reference media datasets. - Determine an indication of a trained machine learning model’s capability to correctly verify the object in the media dataset. - Perform an action based on the indication.
The inventive features center on a system and methods for dynamically assessing a machine learning model's capability to verify objects in media datasets by analyzing the presence of specific operating conditions and comparing them to a pool of labeled reference conditions, enabling suitable actions based on model robustness.
Stated Advantages
Allows assessment of machine learning model capabilities under varying operating conditions, providing an operator with a confidence level and an indication of the quality of object detection.
Enables identification of operating condition limits for model performance, revealing the scope of conditions under which the model can accurately function.
Provides feedback about specific operating conditions that affect detection, facilitating targeted improvements.
Supports suggestions for improving the reference dataset pool, enhancing calibration and verification processes compared to systems that do not evaluate or recommend additional reference data.
Documented Applications
Forensic speech analysis, including automatic assessment of speaker-intrinsic and speaker-extrinsic conditions and evaluating expected performance in forensic speaker recognition.
Interested in licensing this patent?