What we do, just is what we know

Who are you?
Over my next few blogs I’m going to talk a bit more about what I’m doing and how.  I hope it’s of some interest to people in itself, but I also want to use it to illustrate some wider points about data collection.

Last time I talked about learning analytics, and gave one example of a difficulty in measuring whatever outcome variable we’re looking at.  I probably should also have highlighted a distinction sometimes made between outcomes - broad impact, and outputs - the products produced by certain actions.  The former could represent the improvement in quality of life £5 would give, while the latter is that £5, perhaps the products it goes into, etc.

This time I want to broaden the picture a bit more and look at my own study, so we'll have 4 posts over the next 2 weeks:

1) Overview of epistemology
2) Overview of how what we're looking for/what we think's out there might influence methods
3) An introduction to the literature I'm looking at on information literacy and access
4) Some thoughts on my practical experiences of field work (and preliminary outcomes!)

Tell me, who are you?
Last week I talked about one of the problems with measuring ‘success’ in search engine tasks - the ways in which we might define success are bound up in a set of assumptions.  The way we chose to operationalise variables is likely to be related the purpose of the variable in fairly important ways.  We can draw out 4 broad approaches:

• Structuralist, realist, positivists – more likely to look for facts about the real world, for example an interest in the things people search for.  For example, these approaches might have an interest in demographic factors about NEETS, etc.
• Individualist, constructivist approaches, focus on cognition and interpretivism  – look for the ways in which individuals come to know and understand the world.  However, they can be accused of focussing too much on individuals at cost of understanding their sociocultural setting.  Sometimes seen as idealist  – asserting that the world is mentally constructed. 
• Affective, subjectivist, postmodern approaches – interested in understanding how individuals see meaning, and subjective factors.  Broadly take the view that there is no ‘one’ reality.
• Pragmatist sociocultural approaches – the world is understood through action, we are perceiver-actors and although there might be a ‘real world’ out there, our access to it is mediated by both biology and culture.

Other people might play on these distinction to a greater or lesser degree than me.  However, they do highlight that a useful point to do with how what we're looking for, what we think is "out there" might influence how we look for it. 

Broadly, I’m interested in a pragmatist position based on Dewey (1916), and neo-pragmatists such as Rorty (1981) and McDowell (1996).  This approach recognises that consideration of the usefulness of knowledge and language ‘in action’ at work in the world, is preferable to trying to get at the ‘real world’.  So instead of looking for a true reality, or assuming people have a coherent internal image of the world, we look at the ways in which knowledge and language act on and in the world.

So that’s how I see our access to the world. I’ve also found it useful to think of how to investigate that access broadly speaking, following ‘Activity Theory’ which suggests that we view humans as engaged in activity, which involves a mediation between themselves, the activity they're engaged in, artefacts (e.g. the internet, pen & paper), rules, community, and division of labour.   

Understanding the relationship between environment, and subject involves viewing them through the lens of the activity in which they are engaged - often illustrated by an ‘Expanded Triangle of Activity’, in which relations between elements of the activity system are seen as bidirectional (and contradictions – e.g. ‘rules’ restricting sharing of artefacts like notes – are indicated by jagged arrows between elements), centred on the object of activity. 

The Expanded Triangle of Activity for Information Retrieval in Classrooms

Here, the environment – sociocultural and physical – is not simply a force exerted on individuals in activity oriented cognition.  Rather, there is a constant mediation between the two and, as in the pragmatist tradition, individuals can only be understood within their sociocultural context, in which language and its mediating effects are crucial to understanding activity. 

I take it that the best way to think about activity, is as distributed across systems.  So the, “fundamental concept is that cognitive processes are causally both social and neural.  A person is obviously part of society, but causal effects in learning processes may be understood as bidirectional” (Clancey, 2008, p. 12) emphasising that, “1) Knowledge is not passively received but actively built up by the cognizing subject, and 2) the function of cognition is adaptive and serves the organization of the experiential world, not the discovery of ontological reality.” (Clancey, 2008, p. 20). 

I’ve heard some criticisms of it, the most clear - rather positive review - being in a book Fieldwork for Design, in which it is suggested that Activity Theory (and Distributed Cognition) are essentially methodologically agnostic.  I somewhat disagree with that stance, but take the point.  Regardless though, Activity Theory certainly does provide a useful frame for thinking about the ways in which various aspects of a system are brought to bear on activity.  

How can I measure up to anyone now

So, who cares?  Why does this all matter?  Well, I see a few pretty common problems with even very good research, for example:

1. X is common across a group, therefore, x is experienced in the same way by all of that group and its subgroups
2. X is common alongside a phenomenon, therefore x is necessary, or – a bigger leap – causal
3. X is what people say is important/people highlight/people ‘experience’, therefore x is what’s important.

I suppose the first 2 would be more common in quantitative, and the 3^rd in qualitative but that’s not necessarily the case.  So why are these things problems?  Well:

1. Take an obvious example such as ‘mathematics underachievers’. To pick two groups out, the experiences of which are likely to be rather different, those of girls, and white working class boys. 
2. In research the most obvious (and common) examples of this type of claim are in neuroscience to do with active brain regions (there is some concern with fMRI, etc. e.g. the NeuroSkeptic and NeuroCritic blogs) – Cordelia Fine’s book Delusions of Gender talks about this in relation to neurological gender differences (which she claims the evidence is dubious for).  This is also sometimes conflated with the above issue – a failure to acknowledge the systemic factors involved in many phenomenon can, ironically, assume that some other factor is the significant one rather than a key demographic factor – gender differences are one example of this where systemic societal factors may well impact on neurological development, looking at ethnicity (or indeed, gender again) while failing to account for income factors would be another nice example.  The general issue is also irritating in a lot of political commentary, for example the fact that because there’s still a demand for university, top up fees are an acceptable policy to push.
3. For some types of research, obviously the ways in which people experience a phenomenon are important – and it's often interesting to look at what people say, versus other measures such as what they do (or what other people say – for example, pupil self-perception v. teacher-perception).  However, often research focuses on individual affect at the cost of more interesting analysis, and this is often in areas that people will not have thought of in detail previously, may not have tried  to “use” their belief in action, or explored it in great detail. 

I talked about some of this in my last blog and commented on Kieron’s blog.  The reason these things matter is because what we’re aiming for, influences how we try to get there.  If we want to understand pupil experiences of unemployment, we can ask them, we might also want to look at demographic factors to try and fill in the picture and start to understand how experiences vary amongst groups.  What I like about Activity Theory is that it encourages us to think about the systems within which people act.  Practically speaking, that’s about the ways in which those systems influence people’s participation in society. 

Refs

Activity Theory wikipedia page

Intro to research paradigms at Southampton uni

Pragmatism wikipedia page

Dewey, J. (1916). Democracy and education: an introduction to the philosophy of education. Macmillan.

Mcdowell, J. (1996). Mind and World (New ed.). Harvard University Press.

Rorty, R. (1981). PHILOSOPHY AND THE MIRROR OF NATURE. Princeton University Press.

Sundin, O., & Johannisson, J. (2005). The Instrumentality of Information Needs and Relevance. In F. Crestani & I. Ruthven (Eds.), Context: Nature, Impact, and Role (Vol. 3507, pp. 107–118). Berlin, Heidelberg: Springer Berlin Heidelberg. Retrieved from http://www.springerlink.com/content/gb5gfmg8xn8u3gtx/