MICS has recently produced a science brief titled “Guiding Principles for Assessing the Impacts of Citizen Science”. This science brief presents six recommendations for developing the MICS comprehensive Citizen Science Impact Assessment framework that will help overcome the dispersion of approaches in assessing citizen science impacts.
The brief is available to download here.
The recommendations are based on a systematic review of impact assessment approaches, and empirical insights from past and ongoing projects in the field of citizen science. Led by IHE Delft, the MICS project team analysed 77 peer-reviewed publications as well as ten past and ongoing citizen-science projects to investigate how the impact of these projects is typically assessed. The results of this analysis were recently published in the Sustainability Science journal [https://doi.org/10.1007/s11625-021-00959-2].
The MICS UK case study kicked off 2021 by inviting citizen scientists and the project coordinators involved in Outfall Safari to a series of virtual workshop to discuss their thoughts and understanding regarding the ‘impacts’ of the project.
What is Outfall Safari?
Outfall Safari is an innovative citizen science method for locating, assessing the impact of, and reporting on polluted surface water outfalls. Polluting surface water outfalls often occur when household appliances are incorrectly plumbed, ‘misconnected’, into surface water drains, which flow directly into rivers. Citizens use an app to score outfalls based on the appearance and flow. The polluting outfall scores are reported to local water companies who work to trace misconnected pipes to remedy pollution hotspots.
Since Outfall Safari began in 2016 over 200 citizen scientists have been involved in surveying over 150 km of rivers across Greater London. The project helps raise awareness of misconnections, collect valuable data and helps water companies target efforts to reduce pollution and improve our rivers.
More information about Outfall Safari can be found here: Outfall Safari Guide and Resource Pack.
Outfall Safari is used to locate and assess the impact of polluting surface water outfalls. Source: Zoological Society of London (ZSL), 2017.
The workshop series
A series of 3 virtual workshops were held to explore the impacts of Outfall Safari with citizen scientists and coordinators involved in the project. Despite not being able to meet in-person all three workshops received good attendance and we were able to hold some interesting discussions regarding project impact!
Creating an Impact Journey for Outfall Safari
The first 2 workshops entailed activities designed to develop an ‘impact journey’. An impact journey is essentially a map that details the impacts of a project and the actions or steps that contribute to achieving them. Separate workshops were held with citizen scientists and project coordinators to get both viewpoints on the impacts of Outfall Safari. The MICS project explores the impact of citizen science activities on five domains – Environment, Governance, Science & Technology, Society, and Economy – and participants were asked to link impacts to one or more of these domains.
Following these workshops, the MICS project partners synthesised the impact journey maps, grouping impacts under broad headings.
Six long-term impacts of the Outfall Safari project were identified:
Monitoring the impacts of Outfall Safari
The final workshop in the series brought together citizen scientists and project coordinators to agree upon a combined and simplified version of the impact journey, and to identify which impacts were of priority to monitor. We then discussed how these selected impacts could be monitored, who could be involved and what resources may be required.
Over the coming months the MICS project hopes to help the coordinators of Outfall Safari formulate an impact monitoring program for their project.
One of the key requirements for realising the potential of citizen science is evidence and demonstration of its impact and value. Based on a systematic review of 80 publications, combined with empirical insights from ten past and ongoing projects in the field of citizen science, a paper currently under review will present guidelines for a consolidated citizen-science impact assessment framework to help overcome the dispersion of approaches in assessing citizen science impacts; this comprehensive framework enhances the ease and consistency with which impacts can be captured, as well as the comparability of evolving results across projects. MICS's review is framed according to five distinct, yet interlinked, impact domains (society, economy, environment, science, and governance). Existing citizen-science impact assessment approaches provide assessment guidelines unevenly across the five impact domains, and with only a small number providing concrete indicator-level conceptualisations. The analysis of the results generates a number of salient insights which MICS combines in a set of guiding principles for a consolidated impact assessment framework for citizen-science initiatives.
Moreover, this recent paper has been published on the topic, providing a specific methodology for citizen-science cost-benefit analyses in flood risk management and concrete estimates of saving for a case study in Italy.
Earlier this month, members of the MICS team attended the biennial conferences of the European Citizen Science Association [ECSA Conference 2020]. The event was due to be held in Trieste, Italy but due to the ongoing Covid-19 pandemic the conference was instead held online. The upside to the new virtual format was that many international colleagues from outside the European Union could join the conference without the need for travel including two members of the MICS advisory board, Anne Bowser and Jessie Oliver. In total, there were over 500 registered participants and almost 200 speakers.
During the conference, Luigi Ceccaroni presented the MICS project in a session on evaluation in citizen science. The project was warmly received and several new project coordinators were recruited as beta-testers for our MICSLab initiative [https://mics.tools/lab]. There were also presentations from other members of the MICS team: Uta Wehn presented the WeObserve community of practice on impact, Mohammad Gharesifard presented findings from the Ground Truth 2.0 project about the importance of context in community-based citizen science initiatives, and Bruna Gumiero presented a case study of a citizen science project monitoring nutrient concentrations and E. coli in Bologna.
Our thanks to the ECSA committee and the conference organisers. It was a great opportunity to hear from other citizen science projects and the MICS team have learned a lot. Hopefully, by the time of the next ECSA conference in 2022 the MICS tools will have been launched and will be the talk of the conference!
If you missed the conference this year, but would be interested in joining the European citizen science community you can find more about membership to ECSA here: https://ecsa.citizen-science.net/members/
We’ve been interviewing project coordinators about impact; how they define it, measure it and assess it within their projects. We’ve been collecting and analysing their answers, in an effort to better design the platform to suit their needs. You can read a little bit about the projects we’ve been investigating here: https://mics.tools/lab
We have learnt a great deal from speaking to project coordinators, but here are some of the highlights:
Lesson 1: Impact is hard to define
This will come to no surprise to those who have taken an interest in impact, but even just the concept of “impact” is difficult to define. In some citizen science projects, “impact” isn’t mentioned, and the importance of the project is in knowledge creation. Whereas some coordinators consider any change resulting from their project to be an impact, others only see impacts as long-term indirect effects. While it is not for us to say who is right or wrong, it’s important to consider these opinions when developing the MICS platform.
Lesson 2: We measure impact to learn
As many of you involved in different citizen science projects will know, some projects will have work packages and deliverables dedicated to impact; but many don’t. What we wanted to know, was why does measuring the impact of a project matter in any case? We’ve learnt that there is a lot of crossover between a project’s impact and its evaluation: a thorough impact assessment allows projects to see how effective their methodologies are, where opportunities might be, and how best to promote their activities. It also provides an opportunity for learning, so that future projects can be more impactful and successful.
Lesson 3: Projects use different impact assessment methods (with varying levels of success)
Methods of assessing impact vary hugely across the citizen science projects we interviewed: from having no in-house method and not being aware of any formal methodology in the literature; through having a formal in-house method that is too complicated to actually use practically; to using a strict framework such as the Theory of Change. IHE Delft have been exploring published frameworks in an effort to better understand the literature – we’ll have more on that in the next newsletter. For now, it’s clear that a key task for MICS will be to develop a method that suits both impact newbies and experts; we think we’re up to the challenge!
Lesson 4: Impact is important for our volunteers!
Whilst formal impact assessments are often required by citizen science project funders, a lot of project coordinators consider the citizen scientists to be a key audience interested in a project’s impact. This makes sense in terms of sustaining volunteers’ engagement, and is something for MICS to consider when designing the output of the platform: it has to capture the attention and interest of the average citizen scientist, as well as provide enough information to satisfy more formal funding bodies.
Lesson 5: An impact assessment output will have to be both qualitative and quantitative
We asked citizen science project coordinators, “what would a visual representation of impact look like to you?” and – as you can imagine – the answers were as varied as the projects themselves! Some people think of impact as a score out of 10; some people think of it as a heat map; whereas others think of it as more of a story, or collection of experiences. At MICS, we are taking all of this feedback on board, to design an output that will be useful to as many citizen-science projects – their coordinators and volunteers – as possible.
The MICS team are connecting remotely with the project partners and citizen scientists during the coronavirus lockdown. In work package 4 we have been having regular tele-conference calls with our MICS case study leads to develop co-design sessions for citizen scientists. We are keeping in regular contact with citizen scientists via email and phone. For example, in the Italian case study, we are designing citizen science questionnaires to learn about volunteer’s knowledge of environmental problems that can be completed at home. We are also investigating the possibility of holding remote citizen science co-design workshops using online platforms. Whilst we cannot meet face-to-face with our citizen scientists, the conversations never stop! Communication is a vital tool in any citizen science project whether it’s face-to-face or virtually, talking is really important for the success of any citizen science project.
We, the MICS Team, have kicked off 2020 with a stimulating plenary meeting in Cranfield (UK); and we wanted to share the results of this meeting with you!
What did we cover?
The meeting focused on Work Package 2 - Methods for measuring citizen science impact; Work Package 3 - Toolboxes for methods application, information visualisation and delivery; Work Package 4 - Test-site development and tool validation; and the relationships between these work packages.
What did we learn?
As many of you will be aware, measuring citizen science impact is no simple feat. A project which has positive impacts in one domain, may have negative impacts in another. For example; collecting litter from a local stretch of river may have a hugely positive impact on the environment in that area, but may have a negative environmental impact elsewhere if the litter is simply redistributed to another area, or a negative economic impact if all the citizen scientists involved had to take a day off work. Developing metrics for this complicated interplay between positive and negative impacts across the five MICS domains – society, governance, the economy, the environment, and science and technology – is key to the entire MICS project. The plenary meeting was a great opportunity for us to debate these questions as a consortium and reach some agreement on what we want the MICS impact assessment to look like.
We have made exciting developments in the MICS platform, with the design of the toolbox beginning to take formation. We have recently expanded our team to assist in this development; having recruited an experienced developer to help drive this work forward.
During the plenary meeting we were lucky enough to be able to host citizen scientists from Outfall Safari, a UK citizen science project, who gave us a fascinating insight into what impact means for them at the individual level. Progress has also been made in the other MICS case study sites as we kick-off the co-design of citizen science projects with workshops held around Marzenego River, Italy and Creek Rakos, Hungary.
Development of the methods for measuring impact is underway, with associated progress in toolbox design. Co-design of citizen science projects continues to be a focal point, and we will be updating the MICS website to reflect some of the lessons we have learned from this meeting: watch this space!
The Marzenego river begins its course in the north-east of the Venetian Region. Along its 45 km, the river crosses an extremely heterogeneous territory - characterized by rural, industrial and urban areas – ultimately channelling into the artificial Osellino canal which reaches the Venice Lagoon. As a result, the Marzenego receives water from a dense network of drainage canals, which modify the morphology of the watercourse and put areas surrounding the Marzenego at risk of flooding.
Nature Based Solutions (NBSs) aim to manage both the sustainable use of natural resources to address socio-environmental challenges, and the risk of environmental disaster; providing an integrated approach to conserve, manage and preserve the functionality of natural ecosystems. Along the Marzenego river, NBSs may include the restoration of natural habitats through the widening and remodulation of the riverbed, and the creation of wetlands for nutrient and sediment reduction; promoting biodiversity, reducing flood risk, and providing recreational areas for neighbouring communities.
NBSs are particularly effective when they are developed in a co-participative context, in which volunteers can have the opportunity to express their expectations and needs and be involved in the decision-making processes. Citizen science can further involve citizens by including them in the environmental monitoring of the NBS.
The NBS implemented along the Marzenego river provides a suitable case-study for MICS to evaluate; elucidating the impact of citizen science initiatives in this specific environment. To co-design the citizen science activities, MICS adopts and applies the best practice generated by the Ground Truth 2.0 project. This process has already begun with the first of three workshops, designed to identify and define the project and the environmental monitoring activities to be carried out by citizen scientists.
In December 2019, 40 citizens – including scientists, teachers, environmental experts and public authorities – were introduced to the river restoration project; the concepts of citizen science and NBS; and the MICS project as a whole. Through a series of activities – intended to facilitate an effective co-design of the project - the volunteers contributed their views on the issues surrounding flooding and poor water quality, and their expectations for what the project might achieve. Expectations were summarised as an infographic, and demonstrate increased well-being, increased biodiversity, environmental risk mitigation and social development as key issues in need of addressing.
Based on these expectations, the second workshop will utilize co-design methods to identify useful indicators for each citizen science activity, aimed at monitoring the environmental changes before and after the implementation of NBSs. The third workshop will be dedicated to providing the necessary tools (such as practical kits and apps) for volunteers to begin monitoring.
We’ll keep you updated with our progress following these workshops!
The MICS project launched with a kick-off meeting in January 2019. In July, the team came together again, this time in Romania, to discuss the progress in the project so far and plan our next steps. The MICS workplan sets out four main focuses for the project: 1) developing methods for measuring citizen science impact; 2) creating toolboxes for decision makers, citizens and researchers; 3) establishing four citizen science projects; 4) disseminating the results from the project. During the meeting in Romania, we heard from each of these “work packages”.
IHE Delft leads the team developing the methods for measuring the impacts of Citizen Science initiatives. So far, the team has focused on scoping this effort, creating the methodology for developing the impact assessment framework and planning the testing of the impact assessment framework by the MICS case studies and beyond. Overall, MICS is suggesting a three-pronged approach to capturing the impacts of Citizen Science: 1) by means of Impact Stories (in close collaboration with the work of the WeObserve Community of Practice on value and impact of CS for governance; 2) the MICS Impact Assessment Framework which will be available in due time on the MICS online platform; 3) in-depth case studies. Aside from literature reviews and an empirical workshop at the River Restoration Centre Annual Conference (led by Earthwatch), the MICS Advisory Board has also provided useful feedback on the progress so far.
The River Restoration Centre leads the development of four citizen science case studies in the UK, Romania, Italy and Hungary. During the plenary meeting in July, the MICS team visited the Romania case study to see the creation and connection of the Carasuhat Wetland to the Danube River, and hear from local citizens about the project. They explained how the wetland has created new habitats and is supporting sustainable ecotourism activities. These citizens are now going to be involved in the co-design of citizen science activities to monitor environmental quality of the wetland. This was a great trip and highlighted the importance of citizen involvement in the development and monitoring of nature-based solutions.
The next challenge for the project will be initiating the co-design of the citizen science activities at our four case-study sites. Successful co-design is not always easy but we will follow best practice, using a shortened version of the approach from the GroundTruth2.0 project (led by IHE Delft) which established six citizen observatories across Europe and Africa. We will also continue to develop the online toolbox, incorporating the work already done on the MICS approach to impact assessment.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 824711.
MICS science is open science. All the information on this website is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.