Data curation system with version control for workflow states and provenance
Inventors
PEREGRINE, Vladimir Gluzman • Ilyas, Ihab F. • Stonebraker, Michael Ralph • Zdonik, Stan • Palmer, Andrew H. • Pagan, Alexander Richter • Bruckner, Daniel Meir • Beskales, George • Turmukhametova, Aizana • Zhu, Tianyu • Kshetri, Kanak • Liu, Jason • Bates-Haus, Nikolaus
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-11042523-B2
Publication Date
2021-06-22
Expiration Date
Abstract
A data curation system is provided that includes various methods to enable efficient reuse of human and machine effort. To reuse effort, various facilities are presented that model, save, and allow for querying of provenance and state information of a curation workflow and allow for incremental, stateful transitions of the data and metadata thereof.
Core Innovation
The invention provides a method for data curation of a data source with version control, where the data source includes one or more records and each record is a collection of key-value pairs. A current curation state is defined as a parent state of the data source, and the parent state includes the data source with the key-value pairs, facts about which objects in the data source are linked, a linkage system state, questions posed to data experts, and answers received from the data experts. The current curation state is written to a datastore.
The method automatically performs a curation data action on the parent state using a processor, and the curation data action can include transformation of the data source, linkage by adding object linkage facts, linkage by adding linkage questions, linkage by accepting linkage opinions, or data loading by loading a new data source or adding data to an existing source. A child state of the parent state is then automatically created based on the curation data action using a curation process module, thereby creating a new state of the data source.
The method identifies which of the elements would be changed by the curation data action and automatically identifies all of the changes of the elements between the parent state and the child state using the curation process module. Provenance information is identified for the changes between the parent state and the child state, including which curation data actions produced the changes, which conditions have to hold in order for each element to remain valid, or why changes were made. The child state and the provenance information are written to the datastore.
The disclosed system models curation as incremental, stateful transitions, from parent state to child state, with recorded deltas and provenance metadata. It tracks dependencies using provenance DAGs and supports state history operations such as branching, merging, and rebase. The system includes curation process modules for generating curation proposals from machine and expert inputs, creating new states by applying approved changes, and updating and propagating dependencies under configurable dependency processing modes including RESTRICT, PROPAGATE, and OVERRIDE, with structured queries and what-if scenarios.
Claims Coverage
The partial content includes two independent claims (clm-00001 and clm-00012). Each independent claim centers on defining a version-controlled parent curation state, executing an automatically performed curation data action to create a child state, and writing the child state together with provenance information to a datastore; claim clm-00001 additionally specifies validity-conditions provenance, while claim clm-00012 specifies why-changes provenance.
Version-controlled parent and child curation states with datastore persistence
Defining a current curation state as a parent state of the data source to be curated including the data source with records and key-value pairs, facts about which objects are linked, linkage system state, questions posed to data experts, and answers received; writing the parent state to a datastore; automatically creating a child state of the parent state based on a curation data action using a curation process module; and writing the child state and provenance information to the datastore.
Automatic curation data action with transformation, linkage, and data loading
Automatically performing a curation data action on the parent state using a processor, wherein the curation data action includes at least one of: transformation of the data source (in place or creating a new data source); linkage by adding object linkage facts; linkage by adding linkage questions; linkage by accepting linkage opinions; and data loading (loading a new data source or adding data to an existing source).
Element change identification and provenance including producing actions and validity conditions
Identifying which of the elements would be changed by the curation data action; identifying all changes of the elements between the parent state and the child state using the curation process module; and identifying provenance information for the changes including which curation data actions produced the changes and which conditions have to hold in order for each element to remain valid.
Automatic curation data action producing a child state with provenance including why changes were made
Defining a current curation state as a parent state including the data source with records and key-value pairs, facts about which objects are linked, linkage system state, questions posed to data experts, and answers received; writing the parent state to a datastore; automatically performing a curation data action on the parent state and automatically creating a child state based on the curation data action; identifying which elements would be changed and automatically identifying all changes; identifying provenance information for the changes including why the changes were made; and writing the child state and the provenance information to the datastore.
Across both independent claims, the core inventive coverage is a version-controlled curation workflow that stores a parent curation state, automatically executes a curation data action that can involve transformation, linkage, or data loading, creates a child curation state, and identifies what changed and records provenance; claim clm-00001 emphasizes provenance including conditions for validity, while claim clm-00012 emphasizes provenance including why changes were made.
Stated Advantages
Enables version control over curation states with recorded changes and provenance metadata.
Provides dependency tracking using provenance DAGs for updates across curation states.
Supports configurable dependency processing modes including RESTRICT, PROPAGATE, and OVERRIDE.
Supports structured queries and what-if scenarios.
Documented Applications
Curation of a data source containing records of key-value pairs using inputs from a System Operator and Data Experts.
Managing and querying curation states with what-if scenarios.
Dependency-consistent update propagation across curation states using a provenance DAG and dependency processing modes.
Interested in licensing this patent?