Method and computer program product for producing a record clustering with estimated clustering accuracy metrics with confidence intervals
Inventors
Beskales, George Anwar Dany • Batchelor, Alexandra V. • Long, Brian A.
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11294937-B1
Publication Date
2022-04-05
Expiration Date
Abstract
A method is provided for producing a record clustering with estimated accuracy metrics with confidence intervals. These metrics can be used to determine whether a clustering should be accepted as the output of the system, and whether model training is necessary to meet desired clustering accuracy. A collection of test records is used in the process, wherein each test record is a member of a collection of input records.
Core Innovation
The disclosed invention is an entity-resolution system and workflow for producing a record clustering with estimated accuracy metrics and confidence intervals. It applies a model to a collection of records to produce proposed cluster membership for each record and uses a collection of record changes to revise the input, verified, and test datasets. The system then selects a sample of the revised test records and computes estimated accuracy metrics with confidence intervals from the revised test and verified records.
The workflow addresses limitations in accuracy estimation based on pair-, cluster-, and edit-distance metrics. It produces record clustering accuracy estimates that are record-based precision, record-based recall, and record-based F1 using a partially verified test dataset that is maintained under incremental record updates. This includes incorporating record changes into the input records, revised verified records, and revised test records so that the evaluation remains aligned with evolving clustering outcomes.
To support estimated metrics with confidence intervals, the workflow flags test records that are in need of verification and displays records from proposed clusters and verified clusters whose proposed or verified cluster membership is a related cluster. User indications are received via a user interface display to determine whether displayed records are members of a cluster. The system then computes estimated accuracy metrics for the record clustering with confidence intervals using the proposed clusters and the second revised verified records.
Claims Coverage
The partial content provides two independent claims: a method claim and a computer program product claim. Each includes the same core workflow of revising datasets with record changes, generating proposed clusters, selecting and presenting test records requiring verification, receiving user indications, and computing estimated record clustering accuracy metrics with confidence intervals. The independent claims also incorporate inventive features tied to sampling the test records for verification and computing estimated metrics from revised verified and test records.
Producing record clustering with estimated accuracy metrics with confidence intervals from revised test and verified records
A method or computer program product of producing a record clustering with estimated accuracy metrics with confidence intervals, where record changes are incorporated into input, verified, and test collections to produce revised collections, proposed cluster membership is created using a model, a sample is selected for the test records using the revised verified records, a user provides indications via a user interface display for cluster membership, and estimated accuracy metrics with confidence intervals are computed from the second revised test records using the proposed clusters and the second revised verified records.
Incorporating record changes into input, verified, and test collections to produce revised datasets
The method or computer program product incorporates a collection of record changes, where each record change is an add, update or delete and wherein updates and deletes reference records in the collection of input records, into the collection of input records to produce revised input records, into the collection of verified records to produce first revised verified records, and into the collection of test records to produce first revised test records.
Displaying test records in need of verification and related-cluster records for user indications
The method or computer program product displays to a user, on a user interface display, the test records in the second revised test records that are in need of verification, records in the proposed clusters whose proposed cluster membership is a related cluster, and records in the first revised verified records whose verified cluster membership is a related cluster; the method or computer program product then receives user indications via the user interface display regarding whether displayed records are members of a cluster.
Computing estimated accuracy metrics with confidence intervals using proposed clusters and revised verified records
The method or computer program product computes estimated accuracy metrics for the record clustering with confidence intervals from the second revised test records, using the proposed clusters and the second revised verified records.
Across the two independent claims, the inventive focus is producing record clustering accuracy estimates with confidence intervals by revising input, verified, and test datasets using record changes, generating proposed clusters via a model, sampling test records for verification, presenting in-need-of-verification and related-cluster records for user confirmation, and computing estimated accuracy metrics with confidence intervals from the revised test and verified records.
Stated Advantages
Documented Applications
No documented applications found
Interested in licensing this patent?