The feedback loop that keeps on (Globally) giving

In my last post RCTs should only be part of the story I argued that:

 

So if you’re working in a space where you have lots of pilot projects testing new ideas, and are interested in how you get continuous evaluation of your work (a feedback loop), what type of evaluation should you be using? 

 

Well, here’s where it gets interesting.  The problem with traditional evaluation methods is that they are often static (a snapshot at the beginning and end of the project) and don’t allow for a continual adjustment / refinement of the programme work.  When you have project designs that work iteratively, respond to stakeholder needs and adjust according to performance, you need flexible evaluation methods to go along with that. 

 

Also as Goodhart’s and Cambell’s law tell us, when you try and measure something not only do you risk missing what’s going on , but people start working towards the measure, not the thing that’s supposed to actually be happening (no better example of this than the use of exams in schools).

 

So we need open methods of collecting feedback.  The pragmatists amongst you (thank you for still being here) will be saying “er...ask the people you’re working with?”.  Yes, but that leaves us with masses of qualitative data which is very resource intensive to analyse and so out of the reach of most projects.  This is as far as I had got recently.  Then I found out about the Global Giving Storytelling Project led by the wonderful Marc Maxson (@marcmaxson).  It offers ways of collecting storydata that is rich and open, but can be tagged and quantified to support better analysis of what is happening in a situation. I am still grinning with excitement about the whole thing. There's an article about it in the Stanford Social Innovation Review. 

 

Global Giving have been working with Cognitive Edge to develop an approach to mapping the data that you get from stories, or micro narratives.  Cognitive Edge are responsible for SenseMaker®, a propriety software, which is designed to analyse the responses from small snippets of narratives that come from participants on the projects.  There’s a great overview of it on YouTube

 

What excites me about this approach is that it allows people to openly share what is occurring within their community, can easily become a dynamic source of data (asking someone to tell a story regularly feels a lot different to asking them to fill out a ‘feedback questionnaire’ at the end of every session), and offer the chance to make qualitatively rich data relatively easy to analyse and be used by organisations who have a very limited capacity. 

 

There are challenges of course, not least of which is the financial implications of using propriety software like SenseMaker®, and understanding where and how such a system could fit within your own M & E practices.  

 

I’d love to use this Storytelling approach in the future and will be sure to blog on updates to that.  As always, if anyone has any experience about it please let me know.  For those who are interested in it, Marc has some fascinating posts written as the project has developed on his blog http://chewychunks.wordpress.com/

 

In the meantime if you are interested in using stories in M & E but the Cognitive Edge stuff looks a bit too far down the line.  Can I suggest you check out Most Significant Change methodolgy as a low tech, free but very attractive alternative?  

 

It involves you asking your participants what the most significant change was as a result of their participation in the project, again this can be done regularly so offers a feedback loop.  Rick Davies (one of the creators) has very kindly put the guide to MSC online to be accessed by all. http://mande.co.uk/special-issues/most-significant-change-msc/ it's well worth checking out.