Text: aaa | Text only

Evaluation, on the (Cognitive and Digital) Edge

On: 5th September 2012

How do you make a masterplan for evaluation into a reality? (If you haven’t read the evaluation master plan – you’re missing out - it’s here.)

We’re asking our upcoming Digital Edge project partners to use ‘micro-narrative’ approach to evaluation to understand how their work brings about change (I know – sounds fancy doesn’t it?).  In the spirit of openness this blog is partly so that everyone we’re working with can know what all these strange triangles and requests for stories are trying to do.  On the other hand, for anyone who is thinking of using storytelling evaluation in their work, this might help you on your journey.  For more information on the approach I highly recommend looking at Global Giving’s work here and Marc Maxson’s blog posts here.
They key thing for me is that the ‘signification framework’, used by cognitive edge for participants to say what their story is about,  is not trying to elicit evaluative terms like a lot of questionnaires do e.g. ‘how well do you think such and such thing went?’  Rather it seeks to find out what happened. 
cliff edge
There is a world of things that could influence the success of a project.  For us to have an understanding of specifically what is effecting change, we need as much information as possible about the component parts of our partners’ models of intervention. This informs our approach to scaling and future investment.  
All good so far, but if you let a researcher ask the questions you’ll be there forever (I know, I am one).  So deciding what questions we can ask in our ‘signification framework’ and what to leave out is a tricky business.  This is something we’re still working on and have the first draft out for testing, so if you want to have a look and be involved you are very welcome – drop me a line kieron.kirkland[at]nominettrust.org.uk / @kieronkirkland
A key part of asking the right questions is understanding the central categories of information you want to be asking about.  My starting point for condensing the literature on theories of change on working with young people was to break it down into the 5 ws and a H. If we can get sense of what these experiences are, what they involve, where they happen, when, how and why they affect young people, we’ll have a better understanding of the process of change.  It’s asking a lot, but we’re hoping the questions we’ve developed will give us the building blocks (or data points if you want to be posh) to understand this.  So everyone knows what the questions are trying to uncover, at I’ve broadly outlined their purpose below. 
Firstly, we want to know what is changing. This is addressed through questions around the broad domains that affect people’s live - physical wellbeing, relationships and work/education.  We’ve also broken this down into more specific areas of skill development around work, education, networks and content creation. 
Next we’re looking at who’s involved in the experiences.  This is not just who’s there, but who the most influential people are and who benefits or is harmed by experience the story talk about. 
Where a story happens is also important, for example was it online or offline or what physical spaces were important?  
We’ve tried also to think about the time factor – e.g. is the story about a short term change or something that will affect the young person in the longer term?
Lastly we want to know something about the demographic of the people we’re working with, such as age.  We’ve also included a measure of deprivation designed for a young people to answer (the family affluence scale - find out more about here).  This demographic information allows us to start thinking about what elements are important for success in working with different groups of people. 
So it’s a whistle stop tour, but gives you some context of where the signification framework we’ve designed has come from.  I’ve got great hopes for it as an approach.  The signification framework I’ve outline above doesn’t even consider the type of analysis we can do on the actual content of the stories themselves.  But I’ll save thoughts on that until the whole thing is up and running! 
If you want to know more about this framework, our evaluation approach, or stuff in general (within reason, I’m quite good on Thundercats, but not so good on sport) please drop me a line via email or Twitter @kieronkirkland