Why CATs (Critically Appraised Topics) are the Evidence-Informed Teacher's Best Friend

How many times have you heard : 'There's just no way I have the time to engage in research'.     The development of CATS are important given previous posts have  emphasised the need to embed evidence-informed teacher inquiry into the day-to-day work of the school. This post, I will suggest how Critically Appraised Topics (CATS) can be used in a time-efficient manner to summarise and appraise research

What is a Critically Appraised Topic (CAT)?

A CAT is a structured one-page summary of the results of an evidence-based learning opportunity, and where a pedagogic or pupil issue has stimulated a colleague to:  generate a well-formulated question; appraise the evidence; decide how to use that evidence to address the pedagogic issue or pupil needs (Adapted from Straus at al 2011 p239)

Draft Template for a Critically Appraised Topic

Title	Background question
Foreground question (PICO)	Problem Intervention or action required/anticipated Comparisons Outcome
Pedagogic bottom-line	Concise statement of the best available evidence/answers to the questions
Evidence	Description of evidence:        Type of study             Sample size             Results             Conducted by             When Alternatively – if no ‘research’ evidence is available a statement of relevant stakeholder/school evidence relevant. or articulation of  current best experience – by practitioners.
Commentary	Relevance to current setting         Limitations         Strengths
Reference	Include source of evidence and how accessed
Appraiser	Who conducted the appraisal?
Dissemination	How is it intended that the learning is shared?
Date	Date CAT completed
Review date	Is new evidence likely to become available which would require the CAT to be revised?

Limitations in the use of Critically Appraised Topics

Strauss et al identify a number of limitations in using CATS, including:

• Given the speed CATS are produced, not all the relevant evidence may be accessed.
• As the emphasis is on applying the use of the evidence to a particular setting - the conclusions may not be transferable to other contexts
• Due to the speed of production, there may be errors in the CAT

Despite these weaknesses, Straus et al identify several reasons to continue to produce CATS, including: 

• Writing  a CAT summarises and consolidates learning.
• Many teaching challenges are not unique and we may need to draw upon the learning in the future.
• By sharing the learning from our CATs other colleagues can benefit from this learning, possibly contributing to the development of a professional learning community.
• Most teachers will be new to the use of evidence-based practice techniques and by writing up CATs colleagues will have the the opportunity to refine their skills.

However, the most important aspect of the use of CATs is they allow colleagues to engage in evidence-informed/based practice in a time effective way.  CATs in being time efficient, make it easier for them to be used in settings such as Journal Clubs or the like.

Finally

CATS are great example of how evidence-based teacher inquirers can benefit from cross-disciplinary work, and in particular evidence-based medicine.  And despite educational research being  messy, difficult, provisional, and tentative, there are many opportunities for teachers to learn from other fields and disciplines.    

Reference

Straus, S.E., Glasziou, P., Richardson, W. S. & Haynes, B.R. (2011)  Evidence Based Medicine : How to practice and teach it, (4th edition), Churchill Livingston.

Do all good ideas need to be researched? - A response to David Didau

Recently David Didau posed the question : do all good ideas need to be researched?  In doing so, Didau suggested that rather than researching good ideas, teachers would be better off engaged in a form of  'research'  which involved the following:

Step 1 – have a good idea

Step 2 – try it out with some students

Step 3 – think about what happened – is it worth doing again?

Step 4 – think about why it worked. Maybe dig into the reams of existing research to find out what others think. Come up with a theory which provides a post hoc justification for your good idea’s success.

Step 5 – share your idea with other teachers. Ask them to tell you what they liked and what they didn’t like.

Step 6 – improve the idea.

Step 7 – resist, with all your might, the temptation to slap numbers on to your idea in an attempt to justify why it’s good; this is cargo cult science.

Whilst this approach may appear appealing, the proposed model of 'teacher research' is flawed as it is packed with opportunities for cognitive bias.  Soyer and Hogarth (2015) argue that what we think we have learned from experience may well indeed be wrong.  By using experience as our guide to 'what is a good idea', we will have viewed the past through a number of different filters, examples includes; a focus on outcomes not processes; what our closest and not so closest colleagues tell us; and, our own limited powers of reasoning.  These limitations lead to our interpretations being biased, and as a result our current actions may be misguided. Accordingly, a 'good idea' may be a 'bad idea.'  Nevertheless, if we are aware of those biases we can then use techniques to overcome them.  Soyer and Hogarth identify a number of techniques to overcome these biases which include: seeking evidence to disconfirm your 'good-idea'; looking at situations where  a failure was disguised as a success; studying failure by providing 'safe' environments to discuss what went wrong; provide opportunities for disagreement; pursue prevention by engaging in activities such as pre-mortems.

The Spiral of Inquiry 

Fortunately for teachers wishing to engage in evidence-based teacher inquiry, Timperley, Halbert and Kaser (2014) have developed a process of investigation known as the 'spiral of inquiry', which explicitly acknowledges the potential for cognitive bias.    Timperley et al describe the spiral of inquiry process as a multi-tiered and multi-connected process, which is framed by two central questions :What’s going on for our learners?  How do we know? (p7) and is illustrated in the following figure

The individual phases of the spiral of inquiry

Phase 1 Scan what’s going for our learners

Phase 2 Focus on where our energies will potentially make the most difference

Phase 3 Develop and check-out your hunches of what might be a good idea

Phase 4 How and where can we learn more about what to do?

Phase 5 What can we do differently to make enough of a difference?

Phase 6 Have we made ‘enough’ of a difference ?

Let's now look at the developing a hunch phase in more detail.

Developing a hunch

In this phase the focus is on How are WE contributing to this situation? (p13).  Hunches are often based on intuition and are not grounded in evidence.  In this phases, we must have the confidence to bring these hunches to the fore, and explore the underpinning assumptions and evidence.   However, to explore deeply held views, will require 'emotionally safe learning environments for teachers, so these hunches can be explored without fear.   As such developing hunches is concerned with with: getting values and beliefs about our own teaching practice out into the open; identifying those practices that are within our domain and we can do something about; and finally, it's about checking our assumptions, our evidence, and our initial interpretations for validity before taking the next step.

To conclude
Although the model of  'teacher-research/inquiry' proposed by David Didau may appear attractive, it has fallen into the trap of being both simple and simplistic.  Unfortunately, the model pays little or no attentions to the possibility of cognitive bias, which can distort both what has been learnt from both past-experience and the current process of inquiry.  Fortunately, Timperley et al's 'spiral of inquiry' provides a potentially more effective model of teacher-inquiry.  In doing so,  the model explicitly acknowledges the role of cognitive bias, and provides mechanism by which it may possibly be counter-acted.  Finally, if we want teachers to be evidence-informed practitioners, we must not shy-away from the challenges involved.  Rather than provide overly simple solutions we must help teachers gain the skills and expertise they require for genuine evidence-informed/based teacher inquiry, and which can make a real difference to both professional and student learning.

References

Halbert, J and Kaser, L (2013) Spirals of Inquiry for Equity and Quality, BCPVPA Press, Vancouver. Further information available at www.bcpvpa. bc.ca/node/108.

Kaser, L and Halbert, J (2009) Leadership Mindsets: Innovation and Learning in the Transformation of Schools, Routledge, London.

Soyer, E. and Hogarth, R. (2015) Fooled by Experience : What you think you've learned may be wrong.  A guide to finding out the real lessons.  Harvard Business Review May 2015  pp 73 - 77

Timperley, H (2011) Realizing the Power of Professional Learning, Open University Press, London.

Timperley, H, Halbert, J and Kaser, J. (2014) A framework for transforming learning in schools: Innovation and the spiral of inquiry, Centre for Strategic Education, Victoria

The School Research Lead - Should you prioritise evidence-informed inquiry over R&D

This post seeks to share with School Research Leads some of the key findings of Gu et al's (2014) evaluation of teaching school alliances and the subsequent  implications for the development of a research culture within schools.  In particular, this blog will focus on the implications for balance between research and development and evidence-informed teacher inquiry and joint practice development.  To revisit this interim evaluation is particular timely given that one of my personal take-aways from  researchED New York, was the need for schools to give careful consideration to the balance of research and development compared to evidence-informed practice.

Gu et al's report provides the interim findings of of a two-year study of an evaluation of teaching schools, and  which was designed to gather qualitative and quantitative evidence to help understand the impact and effectiveness of such schools.  The report summarises the findings from visits to 18 case study teaching school alliances, which took place in the summer term of 2012/13.  Interviews were held with a diverse range of individuals with different roles and responsibilities within the individual teaching school alliances.  In addition, interviews were held with strategic partners and schools who had been given support from the teaching school.  The report provides insight into the initial work of the teaching schools against the 'Big Six' strands of the teaching school remit.  However, this blog post will focus on the research and development strand.

Research and Development

Positive developments

A number of positive developments were identified, including:

• Partnerships with HEIs providing promising R&D opportunities
• Opportunities for practitioner research had been strengthened
• The development within seem TSAs of R&D as generally underpinning the 'Big 6' rather than being separate and discrete activity.

Challenges

As always with these initiatives a number of challenges came to the fore, which included.

• Given that many teachers see research as both daunting and time-consuming, there is  need to steer R&D towards evidence-based teacher enquiry and joint practice development.
• Not all TSAs had been able to prioritise R&D
• It was recognised that R&D may not demonstrate immediate impact, and because of the time-consuming nature of R&D some TSAs had not focused on this aspect of the Big 6
• There was frustration at the lack of funding available for some aspects of TSAs R&D work

Implications

So what are the implications for School Research Leads of the report's findings.   For me at least there are four implications worth further consideration.

1. School Research Leads need to have a clear understanding of the differences between R&D and evidence-based teacher inquiry.   I have explored this issue in more detail in a previous post, but the difference between the two approaches is: evidence-based inquiry seeks to build on existing evidence to improve practice; whereas, research is about creating new knowledge. My stance on this issue is as follows: the role of the School Research Lead is help teachers improve not prove.   By focusing on evidence-based teacher inquiry it will make possible to support the vast majority of teachers with an appropriate opportunity for professional learning.  Research and development may only be of interest to a small, though highly committed, minority of colleagues.
2. Given the time and resource pressures which are faced by schools, it is important to try and embed evidence-informed practices into the day to day running of a school.  This could take many forms and could include using the the PICO format to help in the development well-formulated and answerable questions.  given the nature of the school-year, there is often a substantial period of time between discussion, decision and subsequent implementation.  This time could be used to ensure that various sources of evidence are drawn upon to answer well-formulated question, which get to the heart of the matter at hand.
3. Although written for purpose of supporting collaboration across schools, Judy Sebba's and colleagues's report on effective approaches to Joint Practice Development, is essential reading for all School Research Leads.  Many of the approaches which are identified are applicable not just between schools but within schools.  In addition, the work of Timperely, Halbert and Kaser on 'spirals of inquiry' is a valuable companion piece.
4. Finally, the key issue is prioritisation, and having a clear sense of the type of activity which is most likely to have an impact upon colleagues's professional learning, and pupil outcomes.  Big high profile projects in partnership with HEIs may have an appeal, but these activities should only be supported if there are clear mechanisms by which the work of these projects can be embedded within the work of the school, and adds to the long-term capacity and capability of the school.  

To be continued

In future posts the intention is to examine in much more detail what is meant be evidence-based teacher inquiry and Joint Practice Development.  In particular, the intention is to look at literature outside of 'education' to consider whether this has something to offer in the development of both practices.

Reference

The teaching schools evaluation: Emerging issues from the early development of case study teaching school alliances, Research brief, March 2014, Qing Gu, Simon Rea, Robert Hill Lindsey Smethem & John Dunford – The University of Nottingham

Joint practice development (JPD) Schools and academies:  What does the evidence suggest are  effective approaches? Judy Sebba, Phillip Kent, Jo Tregenza, University of Sussex, School of Education and Social Work

A framework for transforming learning in schools: Innovation and the spiral of inquiry, Timperley, H, Halbert, J and Kaser, J. (2014) Centre for Strategic Education, Victoria

The School Research Lead - Evidence-Informed Practice and Humble Inquiry

The motivation to write this post arose from a desire to help school research leads avoid the trap of telling colleagues -  'what the evidence is ... ' or 'what the evidence tells you is ....'  Most colleagues, and especially teachers, have never liked to be told what to do, particularly if they think they already know the answer.  Indeed, instead of telling people what the answer is to a question, learning to ask better questions may lead to the creation and development of more positive relationships.  Given the complexity and challenge of working within schools, we cannot hope to bring about improved outcomes for both our pupils and colleagues unless as Schein (2014) states : ... (we) know how to ask questions and build relationships that are based on mutual respect and the recognition that others know things that we may need to know in order to get the job done.  The remainder of this post will, hopefully humbly, attempt to answer these questions:

• What do we mean by 'humble inquiry'?
• How does humble inquiry differ from other forms of inquiry?
• Why is this difference important for evidence-informed practitioners?
• What strategies can be adopted develop the skills associated with humble inquiry?

What do we mean by humble inquiry?

Schein argues that to achieve this we need to learn a specific form of questioning i.e. 'humble inquiry,' which he defines as ... the fine art of drawing someone out, of asking questions to which you do not already know the answer, of building a relationship based on curiosity and interest in the other person.

Although Schein uses a number of examples, especially from high hazard industries, to make the point of what happens when relationships and communications are not based on humility and trust.  Indeed, in schools the importance of leaders being able to build high quality trusting relations is evidenced by researchers such as Robinson (2011).

How does humble inquiry differ from other forms of inquiry?

Given the central part that disciplined or deliberate inquiry has to be the development of evidence and research informed cultures within schools, differentiating humble inquiry from other forms of inquiry is essential. Schein states  Humble inquiry maximises my curiosity and interest in the other person and minimises bias and preconceptions about the other person.  I want to access my ignorance and ask for information in the least biased and threatening way.  I do not want to lead the other person or put him or her into a position of having to give a socially acceptable response. I want to inquire in the way that will best discover what is really on the other person's mind.  I want others to feel that I accept them, am interested in them, and am genuinely curious about what is on their minds regarding the particular situation we find ourselves in. 

Schein argues humble inquiry differentiates itself from other forms of inquiry - such as diagnostic, confrontational and process-led - by not seeking to influence the mental processes of others.  It's about continually seeking to hear what the other person has to say without allowing our own expectations, prejudices or biases come to the fore.

Why is this difference important for evidence-informed practitioners?

If you accept my definition of a school centric evidence-informed practice, posted on 6 April 2015, as

Evidence-informed practice involves the making of decisions within schools through the conscientious, explicit, judicious and skilful use of four sources of information:  

• educator's (at all levels of the school) expertise and judgment  
• open and transparent evidence from both within the school and other sources, which is subject to rigorous evaluation 
• a critical analysis and appraisal of the best available scholarly research - both theory and evidence
• the values and preferences of  pupils, educators, parents, employers and other stakeholders who might be affected by the decision.  

then 'humble inquiry' has an integral role in the work of the evidence-informed practitioner.  First, being an evidence-informed practitioner involves accessing the skills and expertise of all members, and at all levels, of a school community, and is best done in a non-biased or non-threatening manner.   Second, evidence-informed practice has to take into account the values and preferences of pupils, fellow educators etc, and this will require a genuine interest, curiosity and respect for those individuals's values and preferences.

What strategies can be adopted develop the skills associated with humble inquiry?

Schein identifies a number of different strategies which use to develop the attitude of humble inquiry.

Take-time and slow down  - to ensure that you are able to take-stock - to see what is going on - and not miss things through the blur of speedy-action.

Reflect more and ask yourself humble inquiry questions   - before jumping in with 'two-feet' take the time to ask: What's happening here? What do I feel about the situation, and what do I want to get out of it?  What would be the most appropriate action to take?  Who do I need to build a relationship with to make this happen?

Become more mindful - ask the question - what else is happening here?   Try and locate the situation within a broader context.

Try innovating and engage the artist within you - this could be as simple as writing a journal - even maybe blogging about your experiences as a School Research Lead.  Alternatively, it may involve visiting another school or department where they do things in a manner which is different from your normal ways of working.  Exposure to different views is an essential part of the process.

Review and reflect on your own behavior after an event - engage in structured activities such as - After, Action, Review - to get feedback on what happened in a particular situation - making sure you engage with all those who are part of the process.  If something goes wrong, hold meetings to explore what happened, and individuals's  perceptions of events.

Become sensitive to coordination needs in your own work (school) - use humble inquiry to identify current levels of interdependence within a schools, and then work on building relationships that can help with collaboration.

As a leader, build relationships with your team members - recognise that as a leader you are dependent upon your colleagues to make things happen.   In order to make the most of this dependence, it will necessary to build relationships based upon high levels of trust and communication.  Indeed, this may require you as to make yourself in some ways vulnerable to your colleagues.

And finally 

As Daniel Willingham said at researchED New York, "What I try and keep in mind is that I'm actually winning the argument when the other person is talking because I need to know where they're coming from, if I've got any hope of ever convincing them of something." And "It's about planting a seed, walking away, and letting them mull over it on their own time before revisiting it with them."

References

Robinson, V. (2011) Student-Centered Leadership, Jossey-Bass, San Francisco

Schein, E. (2013) Humble Inquiry : The gentle art of asking rather than telling, Berrett-Koehler, London

researchED New York - Some Initial Reflections

The longer I work at developing my skills as a edublogger, the more I realise that there's nothing more stimulating than listening to interesting speakers, who have something meaningful to say.  What people say at events continue to take me by surprise with their insights, sense of purpose and often humour.  Over the last weekend I had the huge privilege of taking part in researchED New York and listening to an incredible range of speakers, each with something worth saying their about specialist topics.  Speakers and topics included:

• Daniel Willingham - The challenge of persuading believers
• Catherine Glennon - A deep look into school-based research
• Daisy Christodoulou - Seven myths about education
• Keven Bartle & Helene Galdin-O'Shea - The emergent tole of research in a UK teaching school alliance
• Lucy Crehan - What can 'top-performing' systems teach us
• Mary Whitehouse - Research-informed curriculum design
• Robert Janke - Datababble : Misinformation or scientifically based research

I also had the good fortune of being able to lead a session, where we explored the following issues: the limitations of existing definitions of evidence-informed practice;  a new school and teacher  centric definition of evidence-informed practice; and finally, looked at some practical ways of becoming a better evidence-informed practitioner (and, if anyone is interested, the slides used in the presentation can be found at the following link).

So what are my personal take-aways, apart from a battered credit-card, from researchED New York?

• A theme that emerged from a number of presentations was the need for teachers to become research literate.  Yet do we know what we mean by research literacy (and numeracy)? What does it look like?  How will we know if research literacy been achieved?  If we don't know what we mean by research literacy, and I thank Mary Whitehouse for this idea, how do we know what questions to ask teachers to find out where they are in their understanding of research and its implications for practice.
• A second theme emerged around trying to determining the correct balance between critiquing and using research to improve practice, and actively participating in research or seeking to produce research outcomes. This balance needs to be considered for both individual teachers and the school as whole.  There is a view, which I certainly hold, that it is best for teachers (and schools) to be become better evidence-informed practitioners before becoming active producers of research outcomes.  For me, I find to difficult to see how teachers and schools can become 'researchers' without the skills to critically appraise a whole range of evidence - including academic research and school data.  Robert Janke's presentation  on errors in evidence-based decision-making certainly raised a number of questions about the extent to which we accurately interpret school data and statistics.
• Engaging in evidence-informed practice and research will inevitably involve engaging with colleagues who have a different perspective than yourself, and there is the whole issue of how do you go about getting others to change their values, beliefs and behaviour.  This was the topic of Professor Daniel Willingham's fantastic presentation, and which has real practical value for the School Research Lead.  Professor Willingham suggested a number of strategies and tactics which may help School Research Leads engage in more productive conversations with their peers, including
• Be selective with your battles - sometimes peace is better than being right
• Recognize that the person you are trying to persuade has another belief which they believe is well reasoned.
• Your job of persuasion is not just about evidence
• If the other person is talking, you are winning, as you are finding our more about their point of view
• And above all, be patient 

To conclude; once again the researchED team of Tom Bennett and Helene Galdin-O'Shea have put on a terrific event.  Furthermore, thanks must be extended to Daniel Randolph and staff of Riverdale Country School for being such wonderful hosts, in a quite magnificent setting.  I look forward to the next event, but not the associated credit-card bill.