iKnowHow: building a wiki for the voluntary sector (part six) - launch and how to measure quality

Going live!

On Wednesday we officially launched iKnowHow. After a bit of exciteable panic and a few last minute technical bugs we pressed the button and opened our wiki doors to the sector.

In the two days it's been live, we've awarded 54 Blue Dots for contributions. Three new pages have been created and another 15 pages edited. Lots of people have been spreading the word on Twitter and in the press including our blog post on the Guardian Voluntary Sector Network about our online wiki. So, we're ending the week on a massive high and want to say thank you to everyone who has contributed so far.

But this is the tip of the iceberg. We have two months to prove that a wiki is a sustainable method for managing voluntary sector support materials. Once the initial buzz has died down, will the wiki sit dormant, waiting for editors? How can we encourage day-to-day involvement?

Throughout the project, we've been thinking a lot about engagement. We're very pleased to be working with Blue Dot World, using their Blue Dots as a contributions incentive. But could we be doing more?

Here, our technical partner Mark Barratt of Text Matters shares his thoughts about measuring quality online.

Rating, reputation and the kindness of strangers

How do you tell if an article is useful before you read it? How do you know if a contribution from a particular person is likely to be worth paying attention to? Contribution and contributor rating and reputation are important to help us all sort the information gems from the web chaff.

KnowHow has had a ‘star rating’ system since the earliest days of the site. It’s used on articles, site sections and the wikiable ‘how-to’ guides. Questions we’ve been asking ourselves:

• is the star rating the best way of reflecting what you think about articles?
• If we use a simpler system would more people use it? And if we use the simplest system (the Facebook ‘like’), are we okay with the fact that Facebook ‘owns’ the ratings – and will likely use that data for their own purposes?
• Should we extend the ‘article rating’ system to create ‘people rating’ systems as many sites, including Amazon and Stackoverflow, do?

Our existing star rating system is fairly easy to use: click ‘Rate this page’ and click a star rating from ‘Not useful’ to ‘Essential’. Once a few users have a rated a page the results are shown on the page itself and are used in other ways too. We don’t advertise it widely, but anyone can look up the highest-rate pages on the site.  And when we made our first ‘wikiable’ section – the how-tos – we used user rating scores as a key way of featuring content on the section front page.

iKnowHows are going live with no rating system at all, but we’re working on it. Here are some of the issues we’re grappling with (and could do with your help on):

Is simpler better?

Not many of our visitors use the rating system and we agonise – is it because it’s too complicated? Wikipedia has rolled out a rating system that’s a lot more complicated to use. All sites that try to get feedback have this problem and our response rate seems to be as good as or better than most. We get about 300 new ratings each month, which is useful, but means a very small proportion of our 85,000 or so unique visitors each month are telling us what they think.

We’re pretty sure we could increase the rate of response for good articles by adding a Facebook ‘like’ button to every page. There are problems here though. We’re depending on the kindness of strangers: Facebook actually owns the information about who has liked which page: they will let us use the data but they’ll use it themselves too. And we’re sort-of selling information about our users which increases the power of Facebook.

The very popular http://stackoverflow.com technical answers site has a slightly more sophisticated system which allows you to ‘vote up’ or ‘vote down’ a response. Stackoverflow uses voting as a main method for users to interact with the site, and ties it directly to the ‘reputation’ ranking of contributors who provide technical answers.

Which bring us to…

Can your reputation be measured?

If you contribute to iKnowHow, you’ll get recognition and direct rewards through the Blue Dot system. This gives you (the contributor) tangible rewards from doing something – anything – in the system. As getting people to engage at all is a good way of getting them to engage repeatedly, this is a good thing. Blue Dots doesn’t, however, tell us anything about the value of people’s contribution: just that they contributed.

So we’re thinking hard about linking our rating system to our contributors’ public pages. Currently all members have a public page which says something about them and lists the things they have created on the site (here’s mine). It’s available from the link in the by-line of every article, news item, how-to and iKnowHow article.

What do you think?

What we don’t know (and what other sites’ experience doesn’t tell us) is how you would feel – both as a contributing member of this community and as a user of the site – about seeing a summary of the ratings given to work by a contributor.

Would that be useful? Would you like well-rated contributors to be identified in some way (like Amazon’s ‘Top Customer Reviewers’ page? Or with badges?

Please let us know.

Mark Barratt