It can be argued well and lengthy about what is appropriate when it comes to evaluating the relevance, quality and significance of research work and making it measurable. The selected good reads encompass a range of perspectives, including open access repositories, research impact assessment, research evaluation projects, comprehensive assessment methods, and research grant evaluation.
For once, let’s not start with theory work, but in a very practical way. The “Your Impact” research guide by the University of Illinois at Chicago (UIC) offers comprehensive information on evaluating research impact. It covers various metrics, tools, and methodologies to assess the societal, academic, and economic impacts of research. This guide provides practical advice to researchers, librarians, and administrators on navigating the complex landscape of research evaluation, empowering them to demonstrate the value and significance of their work.
Choose your methods wisely – they might be assessed
Of course, the choice of method always influences the results. And this also applies to the methods used to measure the impact of science. A recent project on evaluating research conducted by RAND Europe aims to improve understanding and methodologies for assessing research quality and impact. Their website offers insights into ongoing projects, publications, and tools related to research evaluation. RAND Europe’s expertise in research evaluation provides valuable insights for policymakers, funding agencies, and research institutions seeking to enhance evaluation practices and inform evidence-based decision-making.
If you are looking for a clear and theoretically sound introduction to the topic of research evaluation, Evaluating Research in Context: A Method for Comprehensive Assessment by Jack Spaapen, Huub Dijstelbloem and Frank Wamelink from 2007 is recommended. The focus is on one thing, as the title suggests: Context. The right context is important if not only publications in journals and their ranking values are to be counted. Contextual consideration is crucial in science impact assessment. Research takes place within diverse fields, each with its own objectives, methodologies, and timelines. Therefore, relying solely on universal indicators may oversimplify the evaluation process and fail to capture the nuances of different disciplines. By accounting for the contextual aspects, such as field-specific metrics, geographic factors, and research goals, a more accurate assessment of impact can be achieved.
Assessment of research should recognise diversity of outputs, practices and activities
At Oikoplus, we work in a number of projects funded or co-funded by Horizon Europe, the European Union’s research and innovation program. This raises a very practical question: How does the EU measure the impact of the projects it (co-)funds? The EU Commission calls its new impact monitoring framework ‘Key Impact Pathways’. A recent working document provides an insight into the various indicators used by the EU Commission to evaluate projects.
Science impact assessment is essential for evaluating the broader influence and value of research.
When it comes to evaluation and measurability, it is obvious to operationalize success in numbers. However, there is no scheme for this operationalization that can represent the different types of scientific practice in a comparable way. Researchers are aware of this. One answer to the problem is the Coalition for Advancing Research Assessment (CoARA). Hundreds of universities, institutes, and scientific institutions have already joined the Coalition, united by the vision “that the assessment of research, researchers and research organisations recognises the diverse outputs, practices and activities that maximise the quality and impact of research. This requires basing assessment primarily on qualitative judgement, for which peer review is central, supported by responsible use of quantitative indicators.”
Research assessment should always consider the indicators used and the specific context of the research being assessed. By adopting a comprehensive and contextual approach to impact assessment, stakeholders can gain a more nuanced understanding of research outcomes, encourage diverse research pathways, and make informed decisions to support the advancement of science and its positive societal impact.
