Measuring Impact: A Summary of an MLA’17 Special Session on Research Assessment Services

July 21, 2017 at 3:48 pm | Posted in Awards and Recognition, Professional Development, Uncategorized | Leave a comment

Sarah Carnes was awarded a scholarship to attend the MLA Annual Meeting. Congratulations on winning a NAHSL Professional Development Award!

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

As other attendees can attest, MLA ‘17 consisted of a truly vast array of sessions. From the plenary speeches to the lightning rounds, from the special sessions to the vendor demonstrations, MLA 17 provided many opportunities for professional development, process improvement, and inspiration. For me, the conference certainly lived up to the theme of “Dream, Dare, Do.”

I especially valued a special session entitled “From Dreaming to Doing: Implementing Research Assessment Services.” This panel presentation and discussion occurred on Tuesday, May 30th and was sponsored by the Medical Informatics Section, Leadership and Management Section, and the Translational Science Collaboration SIG. The presenters, health science librarians from government and academia, shared a variety of lessons learned and best practices from academic and health science libraries that provide impact assessment services. Specifically, the goals of the session were:

  • Discuss types of research impact assessment services
  • Share resources and tools for assessment based activities
  • Describe experiences and lessons learned
  • Examine how research impacts services

As a clinical librarian supporting both clinicians and researchers, I’m very interested in ways that I can improve how we accurately and effectively assess the impact of individuals, groups, organizations, and programs. I could not have hoped for a better panel of experts:

  • US National Institutes of Health Library
  • Northwestern University Feinberg School of Medicine, Galter Library
  • Weill Cornell Medicine, Samuel J. Wood Library
  • University of Alberta Libraries (attending virtually)
  • Washington University in St. Louis, Becker Medical Library
  • University of Minnesota, Health Sciences Library
  • UMass Center for Clinical & Translational Science, UMass Medical School

Terri Wheeler and Michael Bales of the Samuel J. Wood library at Weill Cornell Medicine described the Citation Impact Tool they developed with Paul Albert, Prakash Adekkanattu, and Sarbajit Dutta. The tool “measures the percentile rank of the number of times papers have been cited, at the level of the individual paper, measured against a baseline of other articles in the same field, of the same type (research or review), and published the same year.” Wheeler and Bales explained that they organize this data by period of time, type, discipline, and percentile rankings rather than just by first and last author. This makes the data more useful and meaningful, allowing them to focus on the types of impact needed to inform leadership, support T32 grants, or fulfill an information need of the individual researcher. They did note that they have an advantage that has allowed them to create such a bespoke tool: Weill Cornell’s medical library is organized under the CIO and they have robust IT support to make automated tools. Thankfully, their tool is available on GitHub at http://bit.ly/citationimpact.

Karen Gutzman and Kristi Holmes of the Galter Health Sciences Library at Northeastern have used Clarivate and other applications (Web of Science, SCOPUS, etc.) and have found that with any application you can derive a lot of useful information. They have developed some refined initiatives (rather than an ad hoc response) that has led to a home grown customized citation tool and a more comprehensive bibliometric program that effectively measures the impact of research. They report this data not only to researchers but also to the University as the leadership wants to know about research impact.

Ya-Ling Lu and Chris Belter represented the NIH staff library. They explained that they started up their program from scratch. In fact, Lu was hired because library noticed they were receiving an increasing number of requests for impact evaluation. The library started a more formal program that measures not only the impact of individual papers but of entire grant portfolios.

Sally Gore, of the UMass Center for Clinical & Translational Science, originally conducted “traditional, informal” work evaluating impact to support tenure and promotion applications and for individual’s interests. Then she observed the evaluation of research impact at Washington University and established a program to better capture research impact at UMass that has become extremely popular.

Several panelists commented that this service does pose some obstacles. For one, it requires a lot of time. Kristi Holmes explained that this service is in high demand, so they are always seeking ways to balance research assessment with providing reference services. Sally Gore said that UMass is conducting an assessment of how much time each assessment “job” requires. The librarians from Weill Cornell mitigate this time burden by designating five librarians who share the responsibilities so that it doesn’t overwhelm one individual librarian.

While most of the panelists provide this as a fee-based service, Scott Library and NIH do charge and others did comment that they may move to such a system as a way of controlling the quantity of requests. The NIH representatives explained that a clear delineation of the services and fees keeps customers’ requests within the true scope of need.

The panelists agreed that assessing impact is complex and requires consideration of many questions, including time period and whether to include publications that occurred prior to position at the current organization or school. A better understanding of the customers’ needs and interests is very important. This includes understanding how the impact assessment is being used by the individual and the organization. This process may require letting the customers know what options are available, such as report format and data visualization, especially if the bibliometric service is unfamiliar to the customer. This can prevent inundating customers with unnecessary information while reducing the time burden for librarians. Another consideration is how the information will be delivered for its purpose and intended impact. Delivery methods can include reports, resumes, or even social media.

The panelists reminded the audience that measuring the number of citations can be more indicative of distribution rather than impact. Events such as public health departments adopting research recommendations and employing them in community health initiatives with measurable outcomes can be a more accurate reflection of impact. As one panelist explained, in the last several years, research linking tanning beds near college campuses to both higher risks and high rates of cancer amongst people 20-29 years old led to public health outreach campaigns and the passage of laws, including those raising the age limit, across the country. Clearly, this is a more meaningful impact than only measuring the number of citations.

More information about this session, including some of the resources mentioned by panelists, is available at https://goo.gl/Q63yku.

 

 

Advertisements

Leave a Comment »

RSS feed for comments on this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.
Entries and comments feeds.

%d bloggers like this: