About the Project

• Purpose
• Background
• Rights and Permissions
• Acknowledgements
• Funding
• Credits

Purpose

The Becker Medical Library Model for Assessment of Research Impact model represents a practical, do-it-yourself tool for tracking the impact of biomedical research. The Model includes guidance for quantifying and documenting research impact as well as resources for locating evidence of research impact.

Back

Background

In May 2007, Mae Gordon, Ph.D., a principal investigator from the Ocular Hypertension Treatment Study (OHTS) came to Becker Medical Library to inquire about obtaining a book via interlibrary loan. During the visit Dr. Gordon expressed interest in the bibliometric methods noted on a poster presented at the 2007 Association for Research in Vision and Ophthalmology (ARVO) meeting, titled “The Impact of NEI-Funded Multi-Center Trials: Bibliometric Indications of Dissemination, Acceptance and Implementation of Trial Findings” by Pamela C. Sieving, and asked if Becker Medical Library could perform bibliometric analysis for the publications from OHTS to determine the citation impact of the OHTS publications.

The methodology as used for the OHTS project involved performing citation analysis of 26 OHTS peer-reviewed journal articles using the SCOPUS, Web of Science, and Essential Science Indicators databases. Citation analysis is a commonly used method of assessing research impact performed by examining an individual publication and how often it was cited, if ever, in subsequent publications.

Of the 26 journal articles, a core group of six journal articles demonstrated high rates of cited-bys. They were often cited by subsequent publications (i.e., peer-reviewed journal articles and patents), and in some instances exceeded baseline citation rates as noted on Essential Science Indicators (ESI), a database that provides for assessment of intellectual impact by analyzing the citation rate for a publication against other publications in a particular area of research or multiple areas of research. The high citation rates for the six journal articles sparked interest as to why these publications were so often cited by other peer-reviewed journal articles and patents. Was this indicative of significant findings that may have resulted in translational outcomes? If so, what were those clinical outcomes and how could we uncover them? What other evidence of research impact could we discover by going beyond a mere citation analysis? In order to provide the OHTS research investigator a more robust analysis of research impact, we broadened the scope of our search to incorporate many more of the research tools and strategies at our disposal.

Expanding the Scope

A cursory search of Google and Yahoo! using key terms related to OHTS (i.e., open angle glaucoma, pachymetry, central corneal thickness, elevated intraocular pressure, ocular hypertension), and the title of OHTS (i.e. “OHTS”) yielded findings such as practice guidelines, continuing education guidelines, curriculum guidelines, insurance coverage documents, quality measures guidelines that noted OHTS as supporting documentation. Further analysis using other databases and web resources also revealed additional evidence of research impact attributed to OHTS findings. These materials are not usually indexed by databases, nor are they consistently noted as cited-by publications, much less, citation counts. Given the depth of evidence of research impact that was not revealed during the preliminary citation analysis it was decided to refine the methodology and perform a full scale evaluation.

As a start, we consulted with research investigators (OHTS and non-OHTS), clinicians, a bioinformaticist, and other librarians to gain insight as to the research process from the bench to the bedside. We reviewed not only OHTS but also several other research studies, both bench and clinical, to locate examples of research outputs. Research outputs are defined as the products generated by the research study and products disseminated by the research investigators that discuss or interpret the findings of the research study. The OHTS research outputs served as a pathway for tracking the diffusion of research findings.

Identifying Additional Stages of the Research Process

Three other stages of the research process were identified along with research output: knowledge transfer, clinical implementation and community benefit. It was decided to focus on the first three stages for the purposes of this study. Assessment of community benefit requires a different methodology and will be addressed in a future project. A number of tangible indicators of research impact related to knowledge transfer, clinical implementation and community benefit were identified and grouped to the appropriate stages along with specific examples of each. Indicators are defined as specific, concrete examples that demonstrate research impact. Examples of tangible indicators include “did OHTS findings result in any new or ancillary research studies?” and “were the OHTS findings used as supporting documentation for coverage of pachymetry?” A number of other research outputs, indicators, and examples of research impact were identified, including those not reflected by the OHTS project.

Developing a Model

After creating a preliminary model of assessment, a second analysis of OHTS findings was undertaken to locate evidence of research impact based on the indicators as identified. As a result of expanding the project beyond citation analysis, we discovered that it is possible, at least for the OHTS findings, to locate extensive evidence of research impact beyond cited-by publication counts. While citation analysis provided useful information it did not reveal the full impact of the OHTS findings by demonstrating evidence of knowledge transfer and synthesis into clinical applications. Going beyond citation counts and using other means of assessing and quantifying evidence of research impact demonstrated a more robust and comprehensive perspective of the research impact of the OHTS.

Given the depth of evidence of research impact that was revealed during our comprehensive analysis of OHTS, we decided to refine the methodology used into a standardized framework for assessing the impact of research. The resulting Becker Medical Library Model for Assessment of Research represents a practical, do-it-yourself tool for tracking impact of biomedical research and includes guidance for quantifying and documenting research impact, and resources for locating evidence of research impact. We want to determine:

• Can the Becker model be replicated by others? Is it feasible? Reliable? Sustainable?
• Is the model applicable for bench/basic studies?
• Is the model a useful tool for quantifying and documenting biomedical research impact?
• Is this model a valid means of measuring the value of biomedical research investment?
• Is this model a valid means for assessing the transfer of medical discovery to the bedside?
• Are there other indicators of research output, knowledge transfer or clinical implementation?
• Are there other resources that can be used to locate evidence of impact?
• Are there other means of quantifying research output?

Feedback on how to improve or expand the Becker model is most welcome as well as examples of additional indicators that can be used to document and quantify evidence of biomedical research impact. Ideas for other resources that may be useful for locating evidence of biomedical research impact are also welcome. Please contact Cathy Sarli with feedback/suggestions or if you have any questions. Future plans for the project include developing the Community Benefit section, and revising and expanding the Becker model based on feedback.

Back

Rights and Permissions

Assessing the Impact of Research is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. Users are free to copy, distribute, display, and adapt the content as noted on this website for non-commercial purposes under the following conditions:

• The work must be attributed as follows:
  • Bernard Becker Medical Library, Washington University School of Medicine
    http://becker.wustl.edu/impact/assessment/index.html
• Share Alike – If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

Please contact our Cathy Sarli if you have created a link to our site or have reused or adapted the content and provide us with the URL to your own site.

Back

Acknowledgements

Becker Medical Library:

Bob Engeszer, Betsy Kelly, Will Olmstadt, Angie Rosengarten, and Lucas Steinbeck

Ocular Hypertension Treatment Study Group:

Mae Gordon, Ph.D., Michael Kass, M.D., Cheryl LaRue, and Ellen Long

The following individuals also provided assistance with the project:

Marcus Banks, Ellen Dickman, Susan Fowler, Ruth Lewis, Mike Sarli, Pamela Sieving, and Rosalie Uchanski, Ph.D.

Back

Funding

The Ocular Hypertension Treatment Study (OHTS) was supported by awards from the National Eye Institute, the National Center on Minority Health and Health Disparities, National Institutes of Health (grants EY09341, EY09307), awards to the Department of Ophthalmology and Visual Sciences at Washington University, the NIH Vision Core Grant P30 EY 02687, awards to the Department of Ophthalmology at University of Miami, the NIH Vision Core Grant P30 EY 01480; Merck Research Laboratories, White House Station, New Jersey, and unrestricted grants from Research to Prevent Blindness, Inc., New York, New York.

Back

Credits

Website content: Cathy Sarli, Ellen Dubinsky, and Kristi Holmes, Ph.D.

Website design: Marco Tocco – Rock Solid Branding

 

Back to Top

Last updated: July 30, 2009