Open Learning Analytics and measuring 'success'

Open Learning Analytics - what is it, how can it help?

In my last couple of blogs, I’ve looked a bit at the importance of innovation in platforms and curricula design over content delivery (here), and one example – Edu-Browse – of a possible tool, alongside some other sources for Ed-Tech tools. 

Last night I caught a bit of the Learning Analytics lecture delivered by Simon Buckingham-Shum from the OU, on the subject of ‘Open Learning Analytics’.  I sadly spent most of the lecture trying to maintain connection (stupid PC!) but from what I understand Simon was talking about the possibility of a designed interface and set of standards for creating an integrated and extensible framework for Learning Analytics [described here .pdf].  Basically, this would be a set of tools to bring together data in visual ways to support learning, with designed (and extensible) interfaces or dashboards for students, teachers, administrators and other stakeholders.  I think he used an analogy of a tool for learning which gave a core frame to be modified as required (my sound cut out), which I guess Edu-Browse would also be.

The benefits they’re attributing to this Open Learning Analytics include:

1. Highlighting of risk factors for drop-out, enabling early intervention and dynamic detection of risk factors
2. Personalisation of learning content, and processes
3. Self-enabling qualities of access to one’s own data, and understanding of learning pathways (again, content and processes).

I should highlight that they’re interested not only in outcomes, but in the learning processes, and not only in analysis of quantitative data but in how we can analyse qualitative data – student interactions, for example – within OLA too.

Measuring 'success' - how successful can we be?

I’ll return to some of these issues in a later post and probably to the Edu-Browse idea too.  But for now I’m going to just highlight one area of my own research to give an indication of some of the issues we face in measurement, and the assumptions measurement requires.  My current research is on collaborative success in search engine use.  Think about it, and it quickly becomes apparent that it’s actually pretty hard to measure ‘success’, unless you just look for whether or not people come back to use your search engine over another.  Indeed Dan Russell, a Google employee, blogged about some of these issues recently in the context of how people (and children do this more) search, find a fact, but then don’t really understand it – in some contexts that’s ok, but in others, it would not be proper to say they’ve been ‘successful’.

So to give my example, broadly speaking,  I’d say we can distinguish 4 approaches to ‘success’ in Information Retrieval tasks:

'Data' is not the same as 'a measure'

I’m not going to go into the specifics of my research, but I hope the table highlights some of the nuance in assumptions inherent to selecting methods.  This is important in the context of RCTS (see also Kieron's blog) often people suggest ‘more rigorous’ research should be done.  That’s fine, and I have to say often I agree – there is some awful research out there.  However, in many fields, the measures are not intuitive.  Health is another nice example – if we want ‘improved outcomes’, what does that mean?  Longevity; quality of life; lower costs; prevention over symptom treatment; which symptoms matter most; and so on.

The OLA move, and my own interest, is suggesting that the best way to measure outcomes is likely to be dynamic, (see another of Kieron's) and depend on the context, and outcomes looked for.  That’s why although I think Ed's latest blog, raising the issue of data loss, in particular in the context of closed data driven by ‘policy-oriented-evidence-making’ is right, I also think that we may not be best served by the ‘representational’ analogy which can sometimes lead to over-emphasis on data capture.  The dynamic ways in which processes and outcomes can be captured and understood as bi-directional, activity oriented and within a particular ecology, is interesting and requires deeper understanding of the activities with which people engage, and the purposes of those activities in a wide and narrow context.

Crucially we need some idea - a model - of the sorts of behaviour that might occur, and what they might indicate.  Having the data available is important for charities, but it only matters if they can build a conceptual model to explore the data with.  For example, one of the papers I've looked at recently describes a model (for which no system currently exists) to explore how students conceptualise what 'knowlege' is, by exploring how they move through a system. Then, we can move from gathering data - and encouraging people to open up their data - to creating appropriate measures, and models to explore and to test.

Do comment, or tweet me @sjgknight if you have anything on these issues or anything else raised on the blogs!