Krakuer suggests that intelligence emerges wherever four processes are taking place: representation, inference, strategy and competition.
A field mouse sees a tasty seed, and takes in its surroundings...
...then predicts the outcome of various courses of action based on experience and instinct (which is derived from the cumulative experience of generations of field mice)...
...then chooses a particular course of action...
...then either gets the seed, loses out to a cunning shrew
or gets eaten by an Owl.
Krakuer goes on to present the evidence for what he calls Cognitive Ubiquity. This is the idea that intelligence in this sense is identifiable at multiple levels, from the actions of a human being down to a single neuron within that human's brain: 'individual' intelligent organisims are in fact made up of similarly intelligent elements and systems in which those four components are observably in play. So our digestive system, our stomach, the cells in the wall of our stomach all exhibit these same basic characteristics of intelligence: they form a representation of their environment, 'choose' an action from a range of options, and exist in their current form only because they performed their function more effectively than the competition.
It is tempting to claim that we can extend this spectrum of intelligence beyond the individual human: families, organisations, societies, nations can surely in some sense be described as behaving in a structurally similar way. However, I believe that this train of thought is primarily useful in that it pinpoints some ways in which we, as collections of interconnected individuals, fail to achieve the eloquent, evolved intelligence exhibited by biological systems.
I'd also argue that one of the keys to this failure is our troubled relationship with data: a relationship that might be improved by making more data more open, but only if we attend to some underlying issues, at least some of which can be brought into focus using Krakuer's analysis.
How do the staff, volunteers and trustees of a voluntary organisation 'represent' their work to themselves, to their funders, to the people they support? Data, open or otherwise, will play a part, but this is always coloured by more diffuse information, primarily in the form of personal experience. Being personal, the impact of such experience on what an individual actually 'sees' is likely to be significant: yet, as groups, we struggle to accommodate the "anecdote", often feeling that it is some kind of quaint bartering token up against the 'hard currency' of data (which is numbers, spreadsheets, all that kind of stuff, isn't it...?).
Sadly, if we get off to a bad start, the rest of the process is compromised: a partial, incoherent representation will impede an organisation's ability to draw useful (i.e. predictive, accurate) inferences. More troubling still is the likelihood that an organisation's representation of itself is further distorted by a predetermined strategy: 'policy-based evidence-making' to borrow a phrase from the more cynical (or realistic) corners of the civil service.
The operation of competition is where things get really messy: individuals and groups compete for resources, connections and support, arguing their case through constructing a representation of themselves which they believe to be what their audience wishes to see. Data is trimmed, polished, flattered with language and, where deemed advantageous, seasoned with carefully selected anecdotes: which in turn sets off another compromised, partial process within the target organisation (made up of individuals who all bring their own more or less arbitrary baggage to the party).
All of which adds up to a collective delusion of intelligence: we obscure crucial subtleties through compromise, hacking away at the mess to 'reveal' something solid, objective, unarguable. Success is defined in self-serving terms by those with the most power, confidence or ambition, fatally distorting the inexorably improving process of genuine competition.
I'm being deliberately catastrophic. We are still limping forward as a species, but I believe this to be primarily a testament to the mitigating influence of our finer instincts over the sophistry of our public discourse.
And I believe that the momentum building behind open data has its roots in those finer instincts: an intuitive sense that honest, exhaustive representation is the crucial starting point for a journey towards a truly intelligent society. We are at a soberingly early stage, and the challenges we face are numerous and profound: most pressingly, we must both expand our definition of data and find ever more eloquent, coherent ways of collating and presenting this data to ourselves.
Opening data is merely a first step in highlighting the weaknesses and gaps in our understanding of what we're really looking at. The next steps must now be to address those weaknesses, and work towards a consistent, cumulative representation which builds upwards through the levels in any given hierarchy. A truly intelligent society will display the fractal intelligence of biological systems, in which the processes of representation, inference, strategy and competition can be seen acting at all of these levels, interlocking and combining to form an integral whole.
Split apart a roman cauliflower and you get lots of little roman cauliflowers: an openly intelligent society will be similarly divisible.
If you think I'm talking a load of brassicas, let me know - leave a comment, send me an email or find me on twitter @ejanderton.