SCORE: Systematizing Confidence in Open Research and Evidence
Social and Behavioral Science (SBS) research is used to combat a host of complex challenges in national security, public health, and geopolitics. However, the inability to replicate and reproduce (R&R) many findings either precludes their application or contributes to policies predicated on poor evidence. Tools to quantify the strength of a claim could help decision makers effectively use SBS research.
Two Six Labs, SIFT, and Michigan State University are developing A+ (Assessing SBS Research through Automated Paper Comprehension and Analysis) that will fully automatically generate accurate R&R confidence scores (CS) directly from SBS publications, removing the reliance on manual feature extraction techniques and subject matter expertise. Specifically, through novel textual parsing of papers and construction of knowledge graphs over them, trained over a large corpus of SBS papers, we will automatically extract claims, results, statistical methods, parameters, effect types, experimental methodologies, and other key features directly from SBS papers. Currently, we are extending our reach to COVID-19 datasets to apply A+ approach in order to produce R&R and confidence scores in that domain.