Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
A machine learning approach for evaluating the replicability of research receives a scholarly paper or publication about a research effort, and applies learning models and natural language processing (NLP) to identify replicability of the asserted results. Rule based and machine learning approaches employ models based on preexisting research papers having defined replicability, and compares language features, research parameters and factual assertions extracted from a research paper for evaluation. A pipeline of evaluations and feature extraction analyzes the content of the research paper and renders a prediction value that the described research can be repeated with similar results.
Core Innovation
A method for evaluating replicability of a research paper receives a narrative representation of a research effort, including a natural language description in paragraphs and sentences. The method computes, for each paragraph, a metric of readability, subjectivity and sentiment, and associates the computed metrics with a set of features for each respective paragraph. The narrative representation is analyzed for identifying assertions made by one or more sentences based on extracted natural language features from the sentences.
A model of research claims is generated by training the model based on features defined by the plurality of sentences in a previous narrative representation, and the trained model is applied to determine, for each sentence, if the sentence defines an assertion. For assertion-defining sentences, a score is computed for each of a summary claim, results claim, concrete hypothesis and test specification and the computed scores are associated with the features for each respective sentence. The features for each respective paragraph are combined with the features and the computed scores for each respective sentence in a hierarchical script.
The assertions are compared to established assertions represented by a replicability model of previous narrative representations based on features of both the paragraphs and the plurality of sentences. Based on the comparison, a repeatability score is predicted to indicate a likelihood of replicating the identified assertions, and for each of the assertions a probability indicative of a random effect causing the outcome result is evaluated. The method aggregates, based on the comparisons, each of the evaluated assertions based on a sample size and an effect size on which the outcome was based to compute a replication probability of the respective assertions.
Claims Coverage
The partial content provides three independent claims covering a method, a computing device, and a computer program, each built around extracting paragraph and sentence level natural language features, identifying assertion sentences, scoring claim types, and comparing against an established replicability model to predict repeatability or replication probability. The main inventive features emphasize paragraph readability, subjectivity and sentiment metrics, sentence assertion identification, hierarchical script feature combination, and replicability model comparison, with additional probability, random-effect, sample size, and effect size aggregation in the document research variant.
Receiving a narrative representation and computing paragraph readability, subjectivity and sentiment
Receives a narrative representation of a research effort including paragraphs and sentences, computes for each paragraph a metric of readability, subjectivity and sentiment, and associates the computed metrics with a set of features for each respective paragraph.
Identifying assertions from sentence natural language features and generating a model of research claims
Analyzes the narrative representation to identify assertions made by one or more sentences based on extracted natural language features from the sentences, generating a model of research claims made in a previous narrative representation by training the model based on features defined by the plurality of sentences.
Scoring summary claim, results claim, concrete hypothesis and test specification per sentence and combining features in a hierarchical script
Applies the trained model of research claims to determine for each sentence if the sentence defines an assertion; when so, computes a score for each of a summary claim, results claim, concrete hypothesis and test specification; associates the computed score with features for each respective sentence; and combines features for each respective paragraph with the features and computed scores for each respective sentence in a hierarchical script.
Comparing assertions to established assertions in a replicability model and predicting repeatability
Compares the assertions to established assertions represented by a replicability model of previous narrative representations based on features of both the paragraphs and the plurality of sentences and, based on the comparison, predicts a repeatability score indicative of a likelihood of replicating the identified assertions.
Evaluating random-effect probability and aggregating sample size and effect size to compute replication probability of assertions
Evaluates, for each of the assertions, a probability indicative of a random effect causing the outcome result and aggregates, based on the comparisons, each evaluated assertion based on a sample size and an effect size on which the outcome was based to compute a replication probability of the respective assertions in the narrative representation.
Across the independent claims, the document centers on an ML and NLP style pipeline that computes paragraph readability, subjectivity and sentiment metrics, identifies assertion sentences using sentence-level natural language features and a trained model of research claims, assigns scores to summary, results, concrete hypothesis and test specification per sentence, combines paragraph and sentence features in a hierarchical script, compares assertions to established assertions in a replicability model, and outputs either a repeatability score or assertion-level replication probabilities using evaluated random-effect probability together with sample size and effect size.
Stated Advantages
Not explicitly described in patent.
Documented Applications
Not explicitly described in patent.
Interested in licensing this patent?