Anita E. Bandrowski, Ph.D. - US grants

Project Summary While  standards  in  reporting  of  scientific  methods  are  absolutely  critical  to  producing  reproducible  science,  meeting  such  standards  is  difficult.  Checklists  and  instructions  are  tough  to  follow  often  resulting  in  low  and inconsistent  compliance.  Scientific  journals  and  societies  as  well  as  the  National  Institutes  of  Health  are  now actively proposing general guidelines to address reproducibility issues, particularly in the reporting of methods  (e.g.,  http://www.cell.com/star-­methods),  but  the  trickier  part  will  be  to  train  the  biomedical  community  to  use these standards to effectively improve how scientific methods are communicated. To support new standards in methods reporting, specifically the RRID standard for Rigor and Transparency of  Key Biological Resources, we propose to build Sci-­Score a text mining based tool suite to help authors meet the  standard.  Sci-­Score  will  provide  an  automated  check  on  compliance  with  the  RRID  standard  already  implemented by over 100 journals including Cell, Journal of Neuroscience, and eLife. The innovation behind Sci-­ score is the provision of a score, which can be obtained by individual investigators, which reflects a numerical  validation of the quality of their methods reporting. We posit that the score will serve as a tool that investigators  and journals can use to compete with themselves and each other, or in the very least allow them to see how  close they are to the average in meeting quality requirements.   Recently, our group has developed a text mining algorithm that has now been successfully been used to detect software tools and databases from the SciCrunch Registry in published papers. Digital tools are one of four resource types that the RRID standard identifies. We propose to extend this approach to the other types of entities: antibodies, cell lines and model organisms. Resource identification along with other quality metrics twill be used to train an algorithm to score the overall quality of the methods document. If successful, the tool could be used by editors, reviewers, and investigators to improve the number of RRIDs, therefore the quality of descriptors of key biological resources in published papers. This SBIR project will build a set of algorithms similar to the resource finding pipeline and develop it into an industrial robust and reconfigurable software system. Our Phase I specific aims include to 1) creating gold sets of data for each resource type and training a set of algorithms for each resource type; 2) designing and evaluating the scoring system; 3) designing and evaluating a report generating system based on the previous aims. In Phase II, we will develop a scalable backend infrastructure to serve the needs of scientific publishers and research community.

Project? ?Summary While standards in reporting of scientific methods are absolutely critical to producing reproducible science, meeting such standards is tedious and difficult. Checklists and instructions are tough to follow often resulting in low and inconsistent compliance. Scientific journals and societies as well as the National Institutes of Health are now actively proposing general guidelines to address reproducibility issues, particularly in the reporting of methods,? ?but? ?the? ?trickier? ?part? ?is? ?to? ?train? ?the? ?biomedical? ?community? ?to? ?use? ?these? ?standards? ?to? ?effectively. To support new standards in methods reporting, especially the RRID standard for Rigor and Transparency of Key Biological Resources, we propose to build Sci-Score a text mining based tool suite to help authors meet the standard. Sci-Score will provide an automated check on compliance with the RRID standard already implemented by over 100 journals including Cell, Journal of Neuroscience, and eLife and other Rigor and transparency standards put forward by the NIH. The innovation behind Sci-score is the provision of a score, which can be obtained by individual investigators or journals. This score reflects an aspect of quality of methods reporting. We posit that the score will serve as a tool that investigators can use to compete with themselves? ?and? ?each? ?other,? ?the? ?way? ?they? ?currently? ?compete? ?on? ?metrics? ?of? ?popularity,? ?i.e.,? ?the? ?H-index. In Phase I of this project and before, our group has successfully developed a text mining algorithm that can detect antibodies, cell lines, organisms and digital resources (all 4 RRID types) and has created a preliminary score. We propose to extend this approach to all research inputs, like chemicals and plasmids that are requested as part of Cell press? STAR Methods (http://www.cell.com/star-methods)?. We also propose to build a set of algorithms to detect whether authors discuss the major sources of irreproducibility outlined by NIH, including investigator blinding, proper randomization and sufficient reporting of sex and other biological variables. Resource identification along with other quality metrics will be used to score the quality of scientific methods section text. If successful, the tool could be used by editors, reviewers, and investigators to improve the? ?quality? ?of? ?the? ?scientific? ?paper. Our Phase II specific aims include 1) enhancing and hardening the core natural language processing pipelines to recognize a broader range of sentences in near real time; 2) building a set of modular tools that will be provided for different groups of users to take advantage of the text mining capability we develop in aim 1. At the end of Phase II, we should have a commercially viable product that will be able to be licensed to serve the needs? ?of? ?the? ?publishers? ?and? ?the? ?broader? ?research? ?community.

Project Summary One of the most glaring yet easily addressable gaps in our current scientific workflow and publication system is improving the way that methods are reported, in particular, the lack of key methodological details necessary for interpreting and reproducing a study. Most authors continue to cite the name of the reagent, like an antibody using the vendor, and the city where the vendor is located, but omit the catalog and lot number making antibodies very difficult to track down, thereby reducing reproducibility of the paper. The Resource Identification, RRID, Initiative has successfully implemented a solution to this lack of identification, by asking authors to include a persistent unique identifier (RRID) for each antibody along with a standard syntax that includes the lot information. This syntax is now required in about 100 journals and accepted in at least 400, was accepted into the EQUATOR network of standards and is under consideration by the JATS committee, the NISO standard for journal article metadata. For antibodies, RRIDs are assigned by the AntibodyRegistry.org, which accepts full catalogs from antibody companies and individual antibody records from authors, who are unable to locate the record for the antibody that they used in a study or one which they created in their lab. This process should be made as easy as possible for authors, and through text analysis we have devised a set of tools that should help authors create better records with less work. The AntibodyRegistry.org was created as part of an academic project and it has successfully incorporated millions of antibody records, thousands of which have now been cited in scientific papers using the RRID syntax. The use of RRID is growing, and in order to support the longer term sustainability of the AntibodyRegistry.org, a core community authority for RRIDs, this resource needs to be enhanced to align with the available commercial and non-commercial funding sources. We propose the addition of features, valuable to antibody companies and journals, to improve market intelligence and reporting around antibodies. We also propose to auto-generate antibody entries for authors and curators when submitting/curating an antibody to decrease the time it takes to complete the task, thereby reducing the barrier to entry and cost.