Why CATs (Critically Appraised Topics) are the Evidence-Informed Teacher's Best Friend

How many times have you heard : 'There's just no way I have the time to engage in research'.     The development of CATS are important given previous posts have  emphasised the need to embed evidence-informed teacher inquiry into the day-to-day work of the school. This post, I will suggest how Critically Appraised Topics (CATS) can be used in a time-efficient manner to summarise and appraise research

What is a Critically Appraised Topic (CAT)?

A CAT is a structured one-page summary of the results of an evidence-based learning opportunity, and where a pedagogic or pupil issue has stimulated a colleague to:  generate a well-formulated question; appraise the evidence; decide how to use that evidence to address the pedagogic issue or pupil needs (Adapted from Straus at al 2011 p239)

Draft Template for a Critically Appraised Topic

Title	Background question
Foreground question (PICO)	Problem Intervention or action required/anticipated Comparisons Outcome
Pedagogic bottom-line	Concise statement of the best available evidence/answers to the questions
Evidence	Description of evidence:        Type of study             Sample size             Results             Conducted by             When Alternatively – if no ‘research’ evidence is available a statement of relevant stakeholder/school evidence relevant. or articulation of  current best experience – by practitioners.
Commentary	Relevance to current setting         Limitations         Strengths
Reference	Include source of evidence and how accessed
Appraiser	Who conducted the appraisal?
Dissemination	How is it intended that the learning is shared?
Date	Date CAT completed
Review date	Is new evidence likely to become available which would require the CAT to be revised?

Limitations in the use of Critically Appraised Topics

Strauss et al identify a number of limitations in using CATS, including:

• Given the speed CATS are produced, not all the relevant evidence may be accessed.
• As the emphasis is on applying the use of the evidence to a particular setting - the conclusions may not be transferable to other contexts
• Due to the speed of production, there may be errors in the CAT

Despite these weaknesses, Straus et al identify several reasons to continue to produce CATS, including: 

• Writing  a CAT summarises and consolidates learning.
• Many teaching challenges are not unique and we may need to draw upon the learning in the future.
• By sharing the learning from our CATs other colleagues can benefit from this learning, possibly contributing to the development of a professional learning community.
• Most teachers will be new to the use of evidence-based practice techniques and by writing up CATs colleagues will have the the opportunity to refine their skills.

However, the most important aspect of the use of CATs is they allow colleagues to engage in evidence-informed/based practice in a time effective way.  CATs in being time efficient, make it easier for them to be used in settings such as Journal Clubs or the like.

Finally

CATS are great example of how evidence-based teacher inquirers can benefit from cross-disciplinary work, and in particular evidence-based medicine.  And despite educational research being  messy, difficult, provisional, and tentative, there are many opportunities for teachers to learn from other fields and disciplines.    

Reference

Straus, S.E., Glasziou, P., Richardson, W. S. & Haynes, B.R. (2011)  Evidence Based Medicine : How to practice and teach it, (4th edition), Churchill Livingston.

Do all good ideas need to be researched? - A response to David Didau

Recently David Didau posed the question : do all good ideas need to be researched?  In doing so, Didau suggested that rather than researching good ideas, teachers would be better off engaged in a form of  'research'  which involved the following:

Step 1 – have a good idea

Step 2 – try it out with some students

Step 3 – think about what happened – is it worth doing again?

Step 4 – think about why it worked. Maybe dig into the reams of existing research to find out what others think. Come up with a theory which provides a post hoc justification for your good idea’s success.

Step 5 – share your idea with other teachers. Ask them to tell you what they liked and what they didn’t like.

Step 6 – improve the idea.

Step 7 – resist, with all your might, the temptation to slap numbers on to your idea in an attempt to justify why it’s good; this is cargo cult science.

Whilst this approach may appear appealing, the proposed model of 'teacher research' is flawed as it is packed with opportunities for cognitive bias.  Soyer and Hogarth (2015) argue that what we think we have learned from experience may well indeed be wrong.  By using experience as our guide to 'what is a good idea', we will have viewed the past through a number of different filters, examples includes; a focus on outcomes not processes; what our closest and not so closest colleagues tell us; and, our own limited powers of reasoning.  These limitations lead to our interpretations being biased, and as a result our current actions may be misguided. Accordingly, a 'good idea' may be a 'bad idea.'  Nevertheless, if we are aware of those biases we can then use techniques to overcome them.  Soyer and Hogarth identify a number of techniques to overcome these biases which include: seeking evidence to disconfirm your 'good-idea'; looking at situations where  a failure was disguised as a success; studying failure by providing 'safe' environments to discuss what went wrong; provide opportunities for disagreement; pursue prevention by engaging in activities such as pre-mortems.

The Spiral of Inquiry 

Fortunately for teachers wishing to engage in evidence-based teacher inquiry, Timperley, Halbert and Kaser (2014) have developed a process of investigation known as the 'spiral of inquiry', which explicitly acknowledges the potential for cognitive bias.    Timperley et al describe the spiral of inquiry process as a multi-tiered and multi-connected process, which is framed by two central questions :What’s going on for our learners?  How do we know? (p7) and is illustrated in the following figure

The individual phases of the spiral of inquiry

Phase 1 Scan what’s going for our learners

Phase 2 Focus on where our energies will potentially make the most difference

Phase 3 Develop and check-out your hunches of what might be a good idea

Phase 4 How and where can we learn more about what to do?

Phase 5 What can we do differently to make enough of a difference?

Phase 6 Have we made ‘enough’ of a difference ?

Let's now look at the developing a hunch phase in more detail.

Developing a hunch

In this phase the focus is on How are WE contributing to this situation? (p13).  Hunches are often based on intuition and are not grounded in evidence.  In this phases, we must have the confidence to bring these hunches to the fore, and explore the underpinning assumptions and evidence.   However, to explore deeply held views, will require 'emotionally safe learning environments for teachers, so these hunches can be explored without fear.   As such developing hunches is concerned with with: getting values and beliefs about our own teaching practice out into the open; identifying those practices that are within our domain and we can do something about; and finally, it's about checking our assumptions, our evidence, and our initial interpretations for validity before taking the next step.

To conclude
Although the model of  'teacher-research/inquiry' proposed by David Didau may appear attractive, it has fallen into the trap of being both simple and simplistic.  Unfortunately, the model pays little or no attentions to the possibility of cognitive bias, which can distort both what has been learnt from both past-experience and the current process of inquiry.  Fortunately, Timperley et al's 'spiral of inquiry' provides a potentially more effective model of teacher-inquiry.  In doing so,  the model explicitly acknowledges the role of cognitive bias, and provides mechanism by which it may possibly be counter-acted.  Finally, if we want teachers to be evidence-informed practitioners, we must not shy-away from the challenges involved.  Rather than provide overly simple solutions we must help teachers gain the skills and expertise they require for genuine evidence-informed/based teacher inquiry, and which can make a real difference to both professional and student learning.

References

Halbert, J and Kaser, L (2013) Spirals of Inquiry for Equity and Quality, BCPVPA Press, Vancouver. Further information available at www.bcpvpa. bc.ca/node/108.

Kaser, L and Halbert, J (2009) Leadership Mindsets: Innovation and Learning in the Transformation of Schools, Routledge, London.

Soyer, E. and Hogarth, R. (2015) Fooled by Experience : What you think you've learned may be wrong.  A guide to finding out the real lessons.  Harvard Business Review May 2015  pp 73 - 77

Timperley, H (2011) Realizing the Power of Professional Learning, Open University Press, London.

Timperley, H, Halbert, J and Kaser, J. (2014) A framework for transforming learning in schools: Innovation and the spiral of inquiry, Centre for Strategic Education, Victoria

The School Research Lead - Should you prioritise evidence-informed inquiry over R&D

This post seeks to share with School Research Leads some of the key findings of Gu et al's (2014) evaluation of teaching school alliances and the subsequent  implications for the development of a research culture within schools.  In particular, this blog will focus on the implications for balance between research and development and evidence-informed teacher inquiry and joint practice development.  To revisit this interim evaluation is particular timely given that one of my personal take-aways from  researchED New York, was the need for schools to give careful consideration to the balance of research and development compared to evidence-informed practice.

Gu et al's report provides the interim findings of of a two-year study of an evaluation of teaching schools, and  which was designed to gather qualitative and quantitative evidence to help understand the impact and effectiveness of such schools.  The report summarises the findings from visits to 18 case study teaching school alliances, which took place in the summer term of 2012/13.  Interviews were held with a diverse range of individuals with different roles and responsibilities within the individual teaching school alliances.  In addition, interviews were held with strategic partners and schools who had been given support from the teaching school.  The report provides insight into the initial work of the teaching schools against the 'Big Six' strands of the teaching school remit.  However, this blog post will focus on the research and development strand.

Research and Development

Positive developments

A number of positive developments were identified, including:

• Partnerships with HEIs providing promising R&D opportunities
• Opportunities for practitioner research had been strengthened
• The development within seem TSAs of R&D as generally underpinning the 'Big 6' rather than being separate and discrete activity.

Challenges

As always with these initiatives a number of challenges came to the fore, which included.

• Given that many teachers see research as both daunting and time-consuming, there is  need to steer R&D towards evidence-based teacher enquiry and joint practice development.
• Not all TSAs had been able to prioritise R&D
• It was recognised that R&D may not demonstrate immediate impact, and because of the time-consuming nature of R&D some TSAs had not focused on this aspect of the Big 6
• There was frustration at the lack of funding available for some aspects of TSAs R&D work

Implications

So what are the implications for School Research Leads of the report's findings.   For me at least there are four implications worth further consideration.

1. School Research Leads need to have a clear understanding of the differences between R&D and evidence-based teacher inquiry.   I have explored this issue in more detail in a previous post, but the difference between the two approaches is: evidence-based inquiry seeks to build on existing evidence to improve practice; whereas, research is about creating new knowledge. My stance on this issue is as follows: the role of the School Research Lead is help teachers improve not prove.   By focusing on evidence-based teacher inquiry it will make possible to support the vast majority of teachers with an appropriate opportunity for professional learning.  Research and development may only be of interest to a small, though highly committed, minority of colleagues.
2. Given the time and resource pressures which are faced by schools, it is important to try and embed evidence-informed practices into the day to day running of a school.  This could take many forms and could include using the the PICO format to help in the development well-formulated and answerable questions.  given the nature of the school-year, there is often a substantial period of time between discussion, decision and subsequent implementation.  This time could be used to ensure that various sources of evidence are drawn upon to answer well-formulated question, which get to the heart of the matter at hand.
3. Although written for purpose of supporting collaboration across schools, Judy Sebba's and colleagues's report on effective approaches to Joint Practice Development, is essential reading for all School Research Leads.  Many of the approaches which are identified are applicable not just between schools but within schools.  In addition, the work of Timperely, Halbert and Kaser on 'spirals of inquiry' is a valuable companion piece.
4. Finally, the key issue is prioritisation, and having a clear sense of the type of activity which is most likely to have an impact upon colleagues's professional learning, and pupil outcomes.  Big high profile projects in partnership with HEIs may have an appeal, but these activities should only be supported if there are clear mechanisms by which the work of these projects can be embedded within the work of the school, and adds to the long-term capacity and capability of the school.  

To be continued

In future posts the intention is to examine in much more detail what is meant be evidence-based teacher inquiry and Joint Practice Development.  In particular, the intention is to look at literature outside of 'education' to consider whether this has something to offer in the development of both practices.

Reference

The teaching schools evaluation: Emerging issues from the early development of case study teaching school alliances, Research brief, March 2014, Qing Gu, Simon Rea, Robert Hill Lindsey Smethem & John Dunford – The University of Nottingham

Joint practice development (JPD) Schools and academies:  What does the evidence suggest are  effective approaches? Judy Sebba, Phillip Kent, Jo Tregenza, University of Sussex, School of Education and Social Work

A framework for transforming learning in schools: Innovation and the spiral of inquiry, Timperley, H, Halbert, J and Kaser, J. (2014) Centre for Strategic Education, Victoria

The School Research Lead - Evidence-Informed Practice and Humble Inquiry

The motivation to write this post arose from a desire to help school research leads avoid the trap of telling colleagues -  'what the evidence is ... ' or 'what the evidence tells you is ....'  Most colleagues, and especially teachers, have never liked to be told what to do, particularly if they think they already know the answer.  Indeed, instead of telling people what the answer is to a question, learning to ask better questions may lead to the creation and development of more positive relationships.  Given the complexity and challenge of working within schools, we cannot hope to bring about improved outcomes for both our pupils and colleagues unless as Schein (2014) states : ... (we) know how to ask questions and build relationships that are based on mutual respect and the recognition that others know things that we may need to know in order to get the job done.  The remainder of this post will, hopefully humbly, attempt to answer these questions:

• What do we mean by 'humble inquiry'?
• How does humble inquiry differ from other forms of inquiry?
• Why is this difference important for evidence-informed practitioners?
• What strategies can be adopted develop the skills associated with humble inquiry?

What do we mean by humble inquiry?

Schein argues that to achieve this we need to learn a specific form of questioning i.e. 'humble inquiry,' which he defines as ... the fine art of drawing someone out, of asking questions to which you do not already know the answer, of building a relationship based on curiosity and interest in the other person.

Although Schein uses a number of examples, especially from high hazard industries, to make the point of what happens when relationships and communications are not based on humility and trust.  Indeed, in schools the importance of leaders being able to build high quality trusting relations is evidenced by researchers such as Robinson (2011).

How does humble inquiry differ from other forms of inquiry?

Given the central part that disciplined or deliberate inquiry has to be the development of evidence and research informed cultures within schools, differentiating humble inquiry from other forms of inquiry is essential. Schein states  Humble inquiry maximises my curiosity and interest in the other person and minimises bias and preconceptions about the other person.  I want to access my ignorance and ask for information in the least biased and threatening way.  I do not want to lead the other person or put him or her into a position of having to give a socially acceptable response. I want to inquire in the way that will best discover what is really on the other person's mind.  I want others to feel that I accept them, am interested in them, and am genuinely curious about what is on their minds regarding the particular situation we find ourselves in. 

Schein argues humble inquiry differentiates itself from other forms of inquiry - such as diagnostic, confrontational and process-led - by not seeking to influence the mental processes of others.  It's about continually seeking to hear what the other person has to say without allowing our own expectations, prejudices or biases come to the fore.

Why is this difference important for evidence-informed practitioners?

If you accept my definition of a school centric evidence-informed practice, posted on 6 April 2015, as

Evidence-informed practice involves the making of decisions within schools through the conscientious, explicit, judicious and skilful use of four sources of information:  

• educator's (at all levels of the school) expertise and judgment  
• open and transparent evidence from both within the school and other sources, which is subject to rigorous evaluation 
• a critical analysis and appraisal of the best available scholarly research - both theory and evidence
• the values and preferences of  pupils, educators, parents, employers and other stakeholders who might be affected by the decision.  

then 'humble inquiry' has an integral role in the work of the evidence-informed practitioner.  First, being an evidence-informed practitioner involves accessing the skills and expertise of all members, and at all levels, of a school community, and is best done in a non-biased or non-threatening manner.   Second, evidence-informed practice has to take into account the values and preferences of pupils, fellow educators etc, and this will require a genuine interest, curiosity and respect for those individuals's values and preferences.

What strategies can be adopted develop the skills associated with humble inquiry?

Schein identifies a number of different strategies which use to develop the attitude of humble inquiry.

Take-time and slow down  - to ensure that you are able to take-stock - to see what is going on - and not miss things through the blur of speedy-action.

Reflect more and ask yourself humble inquiry questions   - before jumping in with 'two-feet' take the time to ask: What's happening here? What do I feel about the situation, and what do I want to get out of it?  What would be the most appropriate action to take?  Who do I need to build a relationship with to make this happen?

Become more mindful - ask the question - what else is happening here?   Try and locate the situation within a broader context.

Try innovating and engage the artist within you - this could be as simple as writing a journal - even maybe blogging about your experiences as a School Research Lead.  Alternatively, it may involve visiting another school or department where they do things in a manner which is different from your normal ways of working.  Exposure to different views is an essential part of the process.

Review and reflect on your own behavior after an event - engage in structured activities such as - After, Action, Review - to get feedback on what happened in a particular situation - making sure you engage with all those who are part of the process.  If something goes wrong, hold meetings to explore what happened, and individuals's  perceptions of events.

Become sensitive to coordination needs in your own work (school) - use humble inquiry to identify current levels of interdependence within a schools, and then work on building relationships that can help with collaboration.

As a leader, build relationships with your team members - recognise that as a leader you are dependent upon your colleagues to make things happen.   In order to make the most of this dependence, it will necessary to build relationships based upon high levels of trust and communication.  Indeed, this may require you as to make yourself in some ways vulnerable to your colleagues.

And finally 

As Daniel Willingham said at researchED New York, "What I try and keep in mind is that I'm actually winning the argument when the other person is talking because I need to know where they're coming from, if I've got any hope of ever convincing them of something." And "It's about planting a seed, walking away, and letting them mull over it on their own time before revisiting it with them."

References

Robinson, V. (2011) Student-Centered Leadership, Jossey-Bass, San Francisco

Schein, E. (2013) Humble Inquiry : The gentle art of asking rather than telling, Berrett-Koehler, London

researchED New York - Some Initial Reflections

The longer I work at developing my skills as a edublogger, the more I realise that there's nothing more stimulating than listening to interesting speakers, who have something meaningful to say.  What people say at events continue to take me by surprise with their insights, sense of purpose and often humour.  Over the last weekend I had the huge privilege of taking part in researchED New York and listening to an incredible range of speakers, each with something worth saying their about specialist topics.  Speakers and topics included:

• Daniel Willingham - The challenge of persuading believers
• Catherine Glennon - A deep look into school-based research
• Daisy Christodoulou - Seven myths about education
• Keven Bartle & Helene Galdin-O'Shea - The emergent tole of research in a UK teaching school alliance
• Lucy Crehan - What can 'top-performing' systems teach us
• Mary Whitehouse - Research-informed curriculum design
• Robert Janke - Datababble : Misinformation or scientifically based research

I also had the good fortune of being able to lead a session, where we explored the following issues: the limitations of existing definitions of evidence-informed practice;  a new school and teacher  centric definition of evidence-informed practice; and finally, looked at some practical ways of becoming a better evidence-informed practitioner (and, if anyone is interested, the slides used in the presentation can be found at the following link).

So what are my personal take-aways, apart from a battered credit-card, from researchED New York?

• A theme that emerged from a number of presentations was the need for teachers to become research literate.  Yet do we know what we mean by research literacy (and numeracy)? What does it look like?  How will we know if research literacy been achieved?  If we don't know what we mean by research literacy, and I thank Mary Whitehouse for this idea, how do we know what questions to ask teachers to find out where they are in their understanding of research and its implications for practice.
• A second theme emerged around trying to determining the correct balance between critiquing and using research to improve practice, and actively participating in research or seeking to produce research outcomes. This balance needs to be considered for both individual teachers and the school as whole.  There is a view, which I certainly hold, that it is best for teachers (and schools) to be become better evidence-informed practitioners before becoming active producers of research outcomes.  For me, I find to difficult to see how teachers and schools can become 'researchers' without the skills to critically appraise a whole range of evidence - including academic research and school data.  Robert Janke's presentation  on errors in evidence-based decision-making certainly raised a number of questions about the extent to which we accurately interpret school data and statistics.
• Engaging in evidence-informed practice and research will inevitably involve engaging with colleagues who have a different perspective than yourself, and there is the whole issue of how do you go about getting others to change their values, beliefs and behaviour.  This was the topic of Professor Daniel Willingham's fantastic presentation, and which has real practical value for the School Research Lead.  Professor Willingham suggested a number of strategies and tactics which may help School Research Leads engage in more productive conversations with their peers, including
• Be selective with your battles - sometimes peace is better than being right
• Recognize that the person you are trying to persuade has another belief which they believe is well reasoned.
• Your job of persuasion is not just about evidence
• If the other person is talking, you are winning, as you are finding our more about their point of view
• And above all, be patient 

To conclude; once again the researchED team of Tom Bennett and Helene Galdin-O'Shea have put on a terrific event.  Furthermore, thanks must be extended to Daniel Randolph and staff of Riverdale Country School for being such wonderful hosts, in a quite magnificent setting.  I look forward to the next event, but not the associated credit-card bill.

The School Research Lead and Cognitive Biases

Acknowledging the influence and impact of cognitive biases on evidence-informed practice is essential for the School Research Lead. Why? Well, evidence-informed practice is built on the notion of teachers using their judgment to make decisions which directly or indirectly benefit their pupils and colleagues.  If those decisions result from systematic errors in judgement due to cognitive limitation, motivational factors and the environment, those decisions are less likely to bring about favourable outcomes.   In this post we will examine the work of Daniel Kahneman and other to help us gain a better understanding of cognitive biases.  Finally, we will examine what strategies can be adopted to mitigate the impact of cognitive biases, whilst at the same time develop our expertise as evidence-informed practitioners. 

What do we mean by cognitive biases?

Amos Tversky and Daniel Kahneman introduced the term 'cognitive bias.' It described systematic errors in judgment and decision-making which can be due to cognitive limitations, motivation factors and the environment. Recently Geoff Petty posted a blog which looked at the impact of confirmation bias on evidence-based practice.  Nevertheless, there is a wide range of cognitive biases, other than confirmation bias, which can impact upon evidence-informed practice, and are summarised by Wilke and Mata (2012).

• Confirmation bias - the tendency to selectively search or interpret information in a way that confirms your perceptions or hypotheses.
• Conjunction fallacy - the tendency to assume that specific conditions are more probable than a single general one.
• Endowment effect - the tendency that people often give more value to on an object they already have than they would be pay to acquire it.
• Fundamental attribution error - the tendency to over-emphasize personal factors and underestimate situational factors when explaining other people's behaviour
• Gambler's fallacy - the tendency to think that future probabilities are changed by past events.
• Halo effect - the tendency for a person's positive or negative traits to extend from one are of their personality to others' perceptions of them. 
• Hindsight bias - a memory distortion phenomenon due to the benefit of feedback about the outcome of an event, people's recalled judgment of the likelihood of that event are typically closers to the actual outcome that their original judgments were.
• Illusory correlations - the tendency to identify a correlation between a certain type of action and effect when no such correlation exists.
• In-group bias - the tendency for people to give preferential treatment to others they perceive to be members of their own group.
• Mere exposure effect - the tendency by which people develop a preference for things merely because they are familiar with them.

Let's now look at one of the cognitive biases - the Halo Effect - in more detail.

Cognitive bias and the Halo Effect

Phil Rosenzweig in his book The Halo Effect .... and the eight other business delusions that deceive managers  describes the Halo Effect as when, for example,  a company's sales and profits are up (a school's retention, achievement and success rates) people (inspectors or significant other stakeholders) may conclude that this is the results of brilliant leadership and strategy or a strong and coherent corporate (school) culture. When performance falters, (success rates or position in leagues tables fall) people conclude it is the result of weak leadership and management (the Headteacher/ Principal), and the company was arrogant or complacent (the school was coasting). Whereas the reality may be that little may has changed and that the school/company performance creates a HALO. This shapes the way judgements are made about outcomes for learners, teaching learning and assessment, leadership and management.

What can we do - a check-list for routing out cognitive biases

At the core of evidence-informed practice is the notion of making decisions within schools through the conscientious, explicit, judicious and skilful use of teacher expertise, school evidence, research evidence and values and preferences of stakeholders.  It should be self-evident that 'cognitive bias' is a real-threat to the making of decisions which lead to favourable outcomes

Kahneman, Lavallo and Sibony (2011) provide guidance on how to find dangerous biases before they lead to poor-decision-making.  Accordingly, Kahneman et al suggest you apply the following check-list.

• Is there any reason to suspect that the team or individual making the recommendation are making errors motivated by self-interest?  If so, review the proposal with care and challenge any underpinning assumptions and evidence.  What are the proposers going to 'get out' of the proposal?
• Has the team fallen in love with its proposals? Use this check-list to test the thinking and evidence which lie behind the proposals
• Were there dissenting opinions, were these opinions fully explored? Is the team victim of 'group-think'? If so, discretely seek out dissenting opinions or other sources of evidence 
• Could the diagnosis be overly influenced by an analogy to a memorable success? Has something similar, but different, worked in the past?  For example, remember when we did this, as that worked, so let's do the same 
• Are credible alternatives included with the recommendation? Have alternatives ways of accessing evidence been considered?  Have different ways of conducting  study been considered, be it quantitative and or qualitative?
• If this decision was to be made again in a year’s time, what information would you want and can you get more of it now? Is an internal source of evidence available to help inform the decision, have interim reports of national studies been published?
• Do you know where the numbers came from – are there unsubstantiated numbers – have they been extrapolated from historical data?  Have assumptions been made about the direction of cause and effect, which do not stand up to critical scrutiny?
• Is the team assuming that a person, organisation or innovation which is successful in one area will be just as successful in another? Just because a particular approach worked in say, the science department, does that mean it will work in art or a humanities subject.
• Is the base case overly optimistic? Have the proposers assumed there will be no delays or barriers to implementation?  Most planners underestimate the resources, especially, needed to complete a project.
• Is the worst case bad enough?  Has a pre-mortem been conducted?  Imagine that the project failed, then think back as to what could have contributed to the failure.
• Is the recommending team overly cautious?  Have the proposers under-estimated the potential benefits? Have they down-played some of the positive indirect benefits?

What are the implications for the School Research Lead

Initially, four implications come to mind when thinking about how to minimise the risk of cognitive biases when engaging in evidence-informed practice within a school.  First, make sure the check-list is applied.  It will not get rid of all the potential biases, though it will be a start.  Second, use all of the check-list. Do not cherry pick those bits of the check-list which 'suit' your current stance or position.  Three, separate the recommenders from the decision-makers in the decision-making process. Proposed research projects need to be scrutinised by others, especially for ethical considerations, before approval is given  Finally, remember getting decisions right is an art not a science.  You won't be able to absolutely prove a research proposal is a good idea, but what you can do is increase your chance of success, be aware of the odds and manage the invariable trade-offs.

As for next week, I will be posting from researchED New York, so hope to have some fascinating news, ideas and perspectives to share.

References

Kahneman, D., Lovallo, D and Sibony, O. (2011) Before you make that big decision ... Harvard Business Review, June 2011

Rosenzweig, P. (2007) The Halo Effect .... and eight other business delusions that deceive managers, Free Press, London.

Wilke A. and Mata R. (2012) Cognitive Bias. In: V.S. Ramachandran (ed.) The Encyclopedia of Human Behavior, vol. 1, pp. 531-535. Academic Press.

Research Leads Brighton and keeping groupthink at bay?

researchED have done it again, with another fantastic event.  Last Saturday, Brighton University played host to the third researchED Research Leads conference.  As usual there were speakers who provided a range of insights into the use of evidence to inform practice.  Individuals, who are willing to give up Saturday to engage in real and effective professional learning, included:

• Daniel Muijs - Can we (reliably) measure teacher effectiveness? 
• Nick Rose - Developing Tools for Teacher Inquiry
• James Mannion - Praxis : Professional Development through Research Inquiry
• Louise Bamfied and Paul Foster - Building Research Rich Schools and Alliances
• Rebecca Allen - How can you know what works in your school?
• Lia Commissar - Teachers discussing learning with psychologists and neuroscientists - an online platform

However, there is always a danger when meeting up with like-minded individuals that you reinforce existing ‘biases’ or ‘predilections. and engage in groupthink.   Yet challenging your biases is an essential component of both,  genuine evidence-informed practice and the role of the school research lead.  Challenging biases is an essential part of evidence-informed practice, as it is necessary to actively seek out evidence and opinions, which do not confirm with your views or hypotheses. Second, School Research Leads need to mindful that in the first instance they may be working with volunteers, or shall say 'believers' who are like-minded, and which may result in groupthink leading to poor decisions being made about what should be inquired about, and how that inquiry should take-place.

With the danger of groupthink in mind, Sunstein and Hastie’s 2014 book Wiser: Getting beyond groupthink to make groups smarter, provides a range of insights on why and how group decision-making can go wrong. Sunstein and Hastie identify four problems that groups often run into:

• Amplifying rather than correcting, individual errors of judgment
• Cascade effects, as others follow what others say to do
• Polarising, adopting more extreme positions than the ones they began with
• Emphasising what everybody knows instead of focusing on critical information held by a few.

It doesn’t take a lot of imagination to see how School Research Leads may have to face and managed these four problems. However, help is at hand, and Sunstein and Hastie a number of ways to reduce the chance of failure.

Inquisitive and self-silencing leaders – who refuse to state a view at the outset and allow time and space for other ideas and perspective come to the fore.  In other words, when leading a session avoid starting off with the phrase. Well the evidence says..

Priming critical thinking – encourage the development of social norms where robust discussion is encouraged, try and reduce the incidence of subtle clues which re-inforce silence and the non-disclosure of information.  When conducting meetings leave time for a competing view to emerge, particular if a speedy consensus has been reached on what the evidence-says is ’.

Reward group success – emphasise that individuals will be rewarded on the basis of the success of the group, rather than individual outcomes, as this should provide better access to the thinking of a range of individuals.  

Role assignment – ensure that all individuals within the group are told that each holds relevant and different information to be contributed and relevant to the success of the group.  This is particularly the case in schools, as evidence-informed practice involves accessing information from all members of the school community, including administrators, teaching assistant and site management staff. Ask of everyone Tell me - what do you think?

Perspective changing  - all this require is a simple question.  ‘If we were to ask someone else from a different department, school or sector – what would they think about the proposal or idea.

Sunstein and Hastie also recommend consideration of other more sophisticated though challenging methods to improve group decision-making – devil’s advocacy, red teams and the Delphi method – which I leave for you to explore for yourself.

To conclude, it is great to attend event such as researchED research leads one-day conferences, on the other hand, there is a real-risk that it can lead to existing biases and prejudices being confirmed.  These biases may impact on how school research leads conduct their work within their own schools, and which may contribute to groupthink and poor decisions in the development of a school based evidence-informed culture.  However, there are a number of techniques which can be used to minimise this risk, and which involve the careful and mindful management of groups.  Future posts will explore issues related to this post, such as cognitive bias and Edgar Schein’s concept of ‘humble inquiry’.