Systems and methods for evaluating the error rate of human-generated data
Inventors
Bergen, Leon • Ko, Kenneth • Tran, Pelu
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12266158-B2
Publication Date
2025-04-01
Expiration Date
Abstract
Systems, apparatuses, and methods for more efficiently and effectively determining the accuracy with which a human evaluates a set of data, as this may reduce the error in the assessment of a model's performance. This can be helpful in situations where human inputs are used to confirm the output of a machine learning generated classification and in situations where it is desired to evaluate the accuracy of data that may have been labeled or annotated by a human curator. This may assist in reducing the need for new validation/test data when evaluating a new model. The system and methods described can be used to evaluate the accuracy of a trained Machine Learning (ML) model, and as a result, allow a comparison between models based on different ML algorithms.
Core Innovation
The invention evaluates an error rate of human-generated data used with a machine learning classifier by obtaining a set of pairs of data, where each pair comprises an image and human-generated text describing an aspect of the image. For each pair, an image is evaluated using a trained image classifier and the human-generated text is evaluated using a trained language classifier, and one or more pairs are identified in which there is a discrepancy between the evaluations.
For each identified pair, a human-generated label of the image is obtained, where the human-generated label represents a second text describing the aspect of the image. A joint distribution of the trained image classifier and the trained language classifier evaluations is formed, and a performance characteristic of the trained language classifier is obtained. Based on the human-generated labels, the joint distribution, and the performance characteristic, one or more of an estimate of expected occurrence of each image classification by the image classifier, an error rate in the human-generated text, or a performance of the trained image classifier is generated.
The generation further defines a joint distribution between latent and observed variables of the image using an assumption of conditional independence. A set of initializations for the joint distribution is generated, and for each initialization an expectation operation and a maximization operation are performed. The expectation and maximization operations are repeated until a state of convergence is reached, outputs corresponding to each initialization are identified, and the output corresponding to maximizing the likelihood of observed data is selected.
Claims Coverage
The independent claims are three: clm-00001, clm-00007, and clm-00013. Each independent claim covers the same inventive concept of paired image and human-generated text discrepancy detection combined with a conditional-independence joint distribution and expectation-maximization likelihood maximization to estimate expected occurrences and error/performance metrics.
Evaluating paired image and human-generated text with discrepancy detection
Obtaining a set of pairs of data comprising an image and human-generated text describing an aspect of the image; evaluating the image using a trained image classifier and evaluating the human-generated text using a trained language classifier; identifying one or more pairs for which there is a discrepancy between the evaluations.
Using human-generated image labels as second text for discrepant pairs
For each identified pair, obtaining a human-generated label of the image, where the human-generated label represents a second text describing the aspect of the image; forming a joint distribution of the trained image classifier and the trained language classifier evaluations; obtaining a performance characteristic of the trained language classifier.
Estimating image classification expected occurrences and error/performance via conditional-independence em
Generating one or more of an estimate of expected occurrence of each classification by the image classifier, an error rate in the human-generated text, or a performance of the trained image classifier; defining a joint distribution between latent and observed variables of the image using an assumption of conditional independence; generating initializations; performing expectation and maximization operations repeatedly until convergence; identifying em outputs and selecting the output that maximizes the likelihood of observed data.
Across clm-00001, clm-00007, and clm-00013, the inventive coverage centers on pairing images with human-generated text, finding discrepancies between trained image and trained language evaluations, obtaining second text human labels for discrepant pairs, and estimating expected image-classification occurrences and error/performance using a conditional-independence joint distribution with an expectation-maximization procedure that selects the maximized-likelihood result.
Stated Advantages
determining accuracy of the machine learning classifier's performance at correctly classifying input data
determining accuracy of human's ability to classify data which is used in training of the machine learning classifier
Documented Applications
Not explicitly described in patent.
Interested in licensing this patent?