The school research lead, confirmation bias and unknown unknowns

A few weeks ago I had the privilege of attending Professor Chris Brown's @ChrisBrown1475 inaugural lecture at Portsmouth University.  One of the great things about attending such events is that you get the chance to talk to some very interesting people, for example, Ruth Luzmore a primary headteacher at an inner London all through school.  Now my conversations with Ruth - by now online -  led onto a discussion about known unknowns and unknown unknowns has led me to a re-visit some work I had done on this topic in trying to discover the things that we don't know we don't know.For as US Secretary of Defense Donald Rumseld famously, or should I say infamously, said:

There are things we know that we don’t know. (And) there are known unknowns.  That is to say there are things we know we don’t know.  That is to say they are things that we now know what we don’t know.  But there are also unknown unknowns.  These are things that we do not know that we don’t’ know.’  

Feduzi & Runde (2014) argue that we are particularly bad at looking for things beyond what we already know, in that we are prone to confirmation bias i.e. we look for evidence that will confirm what we already know.  In addition, we also tend to be too conservative in our predictions of likely outcomes, which can lead to a clustering of predictions which tend to be over optimistic.

In order to address these problems associated with cognitive bias Feduzi & Runde (2014) have put forward a technique which seeks to expand the number of possible scenarios and at the same time, look for evidence to support those scenarios.   The process is made up of several steps

• Think of three main scenarios which you can envisage being the outcome of a decision – things improve, pretty much stay the same, or get worse
• Place those three outcomes on a favourability scale  
• Now try and imagine the worst possible scenario, where things don’t just get worse there is a total collapse - in other words a scenario which is 'completely off the-scale' 
• Having imagined this scenario, try and find evidence that might make this worst possible scenario a possibility
• Then try and imagine a scenario where success is beyond your wildest dreams,  then go and search for evidence that would make this scenario a possibility.

Feduzi and Runde argue that by doing this then you are likely to discover information that you did not previously know about, and in doing so, you will have uncovered some unknown unknowns.  This uncovering of unknown unknowns is the product of seeking to discover evidence that confirms alternatives, rather than seeking out evidence which rules out alternatives.  By searching out for information which seeks to confirms alternatives provides a counter-point to confirmation bias associated with disproving hypotheses.

So what are the implications for you the school research lead.

• Recognise that there are things that you don't know that you don't know
• When thinking through scenarios - it's important to engage a range of individuals - so that the constituent components of either total success or failure can be explored.
• Try and use a range of techniques - that can help you mitigate the impact of cognitive biases - be it premortems or decision check-lists 

And finally none of the material I've referred to in this post have their origins in education - maybe if we spent a little more time looking at other fields - be it knowledge management, improvement science or implementation science - may we can get to know a lot more about what others know.