The value of participation in evaluations
A bit of personal history
Ready to make a big changes in the world, at the age of 21 I started my first office job at the Argentinian National Ministry of Education, in a program funded by the Inter-American Development Bank. It was there that I asked myself for the first time: "Is all this public money we are investing of any use?" I would soon learn that this question is answered through evaluations: studies or applied research that, using specific methodologies, aim to understand the effects of a project or, in our case, a public policy. Since then, one of the main threads of my professional career has been to understand if and what works when we invest in projects where people are at the center.
Impact. For the first time I understood that this word, "impact", has a lot of power and, at the same time, is quite difficult to measure. Trying to understand the effectiveness of social projects led me to decide to pursue a master's degree that would allow me to deepen my knowledge in this area. I understood during my studies that, methodologically speaking, quantitative and/or qualitative data can be collected. But ultimately, it is the researcher or evaluator who determines what will be evaluated and how it will be done. Much of what I did up to that point sought to understand what caused what effects, that is, to find relationships between variables. Qualitative methodologies proved to be better suited to answer these kinds of questions.
It was at IFAD's Independent Office of Evaluation that a colleague showed me a methodology that had quite a different approach: giving the interviewees the space to tell a story, their story, from their point of view, and to give it meaning themselves. The Sensemaker methodology was then further explained to me by Steff Deprez, founder of Voices that Count, a specialist in this and other types of participatory research.
At that time I also met Adinda Van Hemelrijk, a total expert in participatory methodologies and with whom we would later officially launch Collaborative Impact. This group of consultants specialized in participatory methodologies for evaluative analysis and strategic support is mainly active in the field of international cooperation.
Why participatory methodologies make a lot of sense
As I learned about participatory methodologies, I began to get answers to some of the questions I had up to that point regarding evaluations.
Who am I as an outsider to decide whether something makes sense to a "target group" or not? How do I know which things are important to them, and which are not?
What happens when we as evaluators are already late, when the project has already closed and perhaps the next cycle is almost underway? How do we integrate the lessons learned from the evidence we generate for a future program?
How do we prevent the amount of valuable material produced by these evaluations from succumbing to the famous "drawering" process: receiving the five hundred page report, putting it in the drawer and never bringing it out again?
How to deal with complexity and use the simplicity of conceptual models to better understand what is going on in a given collective or system?
From "experts" to facilitators
As I understand it, evaluators using participatory methodologies are no longer "experts" or, as the cocoa producers interviewed in the Peruvian Amazon used to call me, "dotoras", but rather a kind of learning companion and coach for project implementers. The evaluator is not the one who knows the most about the project, the knowledge already exists. The evaluator becomes a facilitator of knowledge and collective intelligence, creating information flows so that participants can achieve collective learning together. And, if we are lucky, the process itself can be transformative.
Leaving intellectual colonialism behind, valuing and recognizing the knowledge of everyone involved
The objective of the evaluation is thus to generate spaces for active listening and reflection, where all stakeholders can feel heard and can contribute the knowledge they already have. To think that the "beneficiary populations" do not understand and require foreign knowledge is not only quite colonialist, but also erroneous. Paulo Freire already had an opinion on the matter... I also discovered that participatory methodologies are deeply rooted in the continent where I grew up. Latin America has a strong history of listening to voices and social inclusion from a methodological point of view (... and not always political...).
Less text, more narrative and images
In order for everyone to participate, we need to adapt the language and the narrative. We need to leave out the complicated words and explain in a way that everyone can understand. Here it helps to use images, metaphors and games. From my point of view, we should always work with them because in general, as human beings, we are attracted by stories and beautiful things, and less by long texts and monologues in endless presentations.
Greater appropriation of learnings
The well-accompanied evaluative process is no longer a step in the project cycle, but emerges as the automatic corollary of project implementation - reflection and evaluation - adaptation of the project in a next phase. The evaluative process becomes a cog in the development of the project. Which brings me to the next realization.
Calling a spade a spade: "Is this actually evaluation?"
Many, if not all projects I know of, have a phase where some type of evaluation occurs. Often what happens is that it is done in a closed room by people with decision-making powers, and without much of a methodological framework. This process is not called evaluation, although in my opinion and especially in small projects, it is often a type of evaluation. Participatory methodologies would allow, in this case, to open the doors in decision making to include the voices of stakeholders beyond the decision makers, and would automatically generate learnings that are already integrated into the system in which the project resides. When there is ownership of the learnings, there is a greater chance that they will be applied at a later stage. These faster iterations are very much like agile methodologies: implement and learn fast to be able to adapt to the context.
Inspiration sources
Much, much has been produced that nourishes the school that strives for participatory methodologies. I continuously will add sources in this list of sources of inspiration:
Philosophical bases
Paulo Freire
Websites
Documentary series "Sowing and Harvesting" highly recommended for its content and for the use of videos to show what has been done.
Books
Mertens. Transformative Research and Evaluation
Chambers (2017). Can we know better?