Reading List EN Uncategorized

RL #030: Beyond Comparing Numbers: Qualitative Research Assessment 

It can be argued well and lengthy about what is appropriate when it comes to evaluating the relevance, quality and significance of research work and making it measurable. The selected good reads encompass a range of perspectives, including open access repositories, research impact assessment, research evaluation projects, comprehensive assessment methods, and research grant evaluation. 

For once, let’s not start with theory work, but in a very practical way. The “Your Impact” research guide by the University of Illinois at Chicago (UIC) offers comprehensive information on evaluating research impact. It covers various metrics, tools, and methodologies to assess the societal, academic, and economic impacts of research. This guide provides practical advice to researchers, librarians, and administrators on navigating the complex landscape of research evaluation, empowering them to demonstrate the value and significance of their work.

Choose your methods wisely – they might be assessed

Of course, the choice of method always influences the results. And this also applies to the methods used to measure the impact of science. A recent project on evaluating research conducted by RAND Europe aims to improve understanding and methodologies for assessing research quality and impact. Their website offers insights into ongoing projects, publications, and tools related to research evaluation. RAND Europe’s expertise in research evaluation provides valuable insights for policymakers, funding agencies, and research institutions seeking to enhance evaluation practices and inform evidence-based decision-making.

If you are looking for a clear and theoretically sound introduction to the topic of research evaluation, Evaluating Research in Context: A Method for Comprehensive Assessment by Jack Spaapen, Huub Dijstelbloem and Frank Wamelink from 2007 is recommended. The focus is on one thing, as the title suggests: Context. The right context is important if not only publications in journals and their ranking values are to be counted. Contextual consideration is crucial in science impact assessment. Research takes place within diverse fields, each with its own objectives, methodologies, and timelines. Therefore, relying solely on universal indicators may oversimplify the evaluation process and fail to capture the nuances of different disciplines. By accounting for the contextual aspects, such as field-specific metrics, geographic factors, and research goals, a more accurate assessment of impact can be achieved.

Assessment of research should recognise diversity of outputs, practices and activities

At Oikoplus, we work in a number of projects funded or co-funded by Horizon Europe, the European Union’s research and innovation program. This raises a very practical question: How does the EU measure the impact of the projects it (co-)funds? The EU Commission calls its new impact monitoring framework ‘Key Impact Pathways’. A recent working document provides an insight into the various indicators used by the EU Commission to evaluate projects.

Science impact assessment is essential for evaluating the broader influence and value of research.

When it comes to evaluation and measurability, it is obvious to operationalize success in numbers. However, there is no scheme for this operationalization that can represent the different types of scientific practice in a comparable way. Researchers are aware of this. One answer to the problem is the Coalition for Advancing Research Assessment (CoARA). Hundreds of universities, institutes, and scientific institutions have already joined the Coalition, united by the vision “that the assessment of research, researchers and research organisations recognises the diverse outputs, practices and activities that maximise the quality and impact of research. This requires basing assessment primarily on qualitative judgement, for which peer review is central, supported by responsible use of quantitative indicators.”

Research assessment should always consider the indicators used and the specific context of the research being assessed. By adopting a comprehensive and contextual approach to impact assessment, stakeholders can gain a more nuanced understanding of research outcomes, encourage diverse research pathways, and make informed decisions to support the advancement of science and its positive societal impact.

Reading List EN Uncategorized

RL #029: Cartography as a Place of Exchange Between Citizens and Experts

The ability to understand one’s immediate surroundings has always been an extremely important skill. For this reason, humanity has spent thousands of years developing and perfecting the craft of representing spatial information including routes or landforms. In today’s age of modern technology, however, the amount and variety of information that needs to be mapped are increasing. Nowadays the ability to have a grasp on our surroundings is proving more complex. This reading list will therefore explore how cartography turns out to be useful to facilitate knowledge exchanges and how it can serve as a vehicle for critical thinking.

Explaining Cartography

Cartography is the practice of map-making. Originally cartographers graphically represented spatial or geographical data but are now faced with having to translate diverse figures from multiple sensors and multiple origins. According to Elik Eizenberg in Forbes technology online magazine, we find ourselves swimming in data (and should care about it). As we can’t fully harness all data, the data scientists’ continuation of collecting new data, slowly loses meaning. Mapping, Georg Gartner argues in an article for Ersi, the global leader in geographic information systems, bridges between human users and all this data. It uses visualization to make science approachable to the public, fully unleashing its potential. 

Point cloud of slope failures in Sensuikyo Valley by LIDAR a tool in modern 3d cartography. Source:

From Knowledge Reception to Knowledge Exchange

Empowering citizens to make informed decisions can also have another effect, namely mutual information exchange. Originally cartographers collect data from various measuring tools such as aerial photographs, remote sensing, field observations, or coordinate lists. This data, however, as mentioned by Horizon 2020-funded WeObserve, has a scarce update date due to increased costs and timely data validation procedures. Today, considering the increased complexity of data, cartographers also turn to alternative sources such as citizens.

Interactive exploration of good and bad governments worldwide by GOV DNA. Source:
Interactive exploration of good and bad governments worldwide by GOV DNA. Source:
Interactive exploration of good and bad governments worldwide by GOV DNA. Source:

According to Caroline Anstey for The New York Times, this new shift towards crowdsourcing information is immensely useful to cartography. Citizens provide both quantitative, but also qualitative data often omitted by cartographers. The citizens’ expertise comes from living in one place for a prolonged period of time. Changes in demographics, environment, human relations, or even housing habits are useful to mapping projects as they can translate into policies or planning decisions. To build trust underlying this exchange, cartographers should provide citizens with clear and understandable information.  

Cartography as a Vehicle for Critical Thinking

According to Sukhmani Mantel for The Conversation, visually mapping relations allows information to engage multiple senses and become relevant to daily life. And indeed, citizens are able to handle novel concepts with an extensive social and cross-cultural understanding. This is what Aleks Buczkowski explains in his piece written for GeoAwesome, the world’s largest geospatial community.

Essentially, Stevenson et al., from Stockholm Environment Institute, claim in an SEI Brief about extreme citizen science approaches in digital mapping, that people from mapping practices, no matter their education level, gain the ability to understand the developing world. This supports their chances to better participate in it also on a more general level: previously excluded groups become aware of how they can co-create and get involved. They now contribute to scientific research, so-called citizen science.

Forensic architecture embedded photographs and videos to reconstruct the story of a single battle during 2014 Gaza War
Forensic architecture embedded photographs and videos to reconstruct the story of a single battle during 2014 Gaza War. Source:

As stated by Fraisl, Heyl, and Hager, researchers at Institute for Applied System Analysis, citizen science is important for the democratization of the scientific field. At the same time, it plays a role in empowering citizens to make informed decisions about their surroundings. This way, as mentioned by Organisation for Security and Co-operation in Europe, authoritative power becomes decentralized and decision-makers can be held accountable for their actions.


Obtaining accurate cartographic data through crowdsourcing is something that is in its early stages, but is increasingly practiced. Especially because now citizens have increasingly more opportunities to use tools, which give them access to global data. On an entrepreneurial scale, this is already taking place. The Domino-E project, which focuses on developing a federation layer optimising the availability of Earth observation data, builds on interoperability and knowledge sharing. Knowledge sharing generates knowledge creation, which is why it is important for cartographers to bet for information exchange as it benefits both them and citizens equally.