The School Research Lead and Stripping and Flipping - Essential skills for the evidence-informed practitioner

For teachers stripping and flipping may not sound like a way of helping them better do their jobs.  Indeed, it may even sound like the road to instant dismissal.  But seriously, stripping and flipping are two essential skills in becoming a research-literate and evidence-informed teacher.  Daniel Willingham in his 2012 book - When Can You Trust The Experts? How to tell good science from bad in education describes how stripping away verbiage and getting to the heart of actual claim being made, is the first step in a 'heuristic' process of evaluating educational interventions, programmes and research.  The rest of this post will now look at stripping and flipping in more detail. But first, lets define a few terms.

The Change refers to a new curriculum or teaching strategy or software package or school restructuring plan - generically anything than someone is urging you to try as a way to better educate kids.

The Persuader refer(s) to any personal who is urging you to try the Change, whether it's a teacher, administrator, salesperson, or the President of the United States (Willingham, p136) 

The Audience refers to individual teachers, department, schools, LEAs who are being asked to adopt the Change

Stripping 

In undertaking an evaluation of a Change Willingham states that it is essential to to be clear on the following:

1. precisely what Change is being suggested
2. precisely what outcome is being promised as a result of the Change
3. the probability that the promised outcome will actually happen if you undertake the Change (Willingham, p136)

For example : If an 'inadequate' school is converted to an academy, there is a 50 percent that the school will double the percentage of pupils gaining 5 GCSEs (inc English and Maths) at grade C or better

Flipping

Flipping is best explained by an example.  Let's suppose that a school claims that 65 percent of pupils leave school at 16 with a least a GCSE grade C in both English and Maths, then 35 percent of pupils leave school without at least a GCSE grade C in both English and Maths.  Now, all other things being equal, due to a framing effects a school with a 65 percent success rate is likely to be rated as better than a school with a 35 percent failure rate.

Things to watch out while flipping and stripping

Of course, there is far more to the stripping and flipping process than can be captured in a couple of examples.  Things to watch of for include"

• Emotional claims about the need or benefit of the Change
• Claims that the Persuader is like you
• False Analogies
• Postive and negative framing
• Old stuff which we have seen before
• Vague stuff which is difficult to discern either the output or outcomes
• Over-reach - claiming the Change will help all teachers and all pupils

Accordingly, there are a number of sub-steps to the Strip It/Flip It process and which are summarised in Table 1

Table 1 : Summary of the Strip It and Flip It Process

Suggested Action	Why You’re Doing This
Strip to the form “If I do X, then there is a Y percent chance of Z happening.”	To get rid of emotional appeals, peripheral cues, and proffered analogies that may influence your belief.  The scientific methods is suppose to be evidence-based and uninfluenced by these factors.
Consider whether the outcome (Z) has an inverse; if so, restate the stripped version of the claim using the inverse	To be sure that you appreciate all consequences of the action – for example, that an 85 percent pass rate implies a 15 percent failure rate.  We are subject to framing effects; we think something is better if the positive aspects are emphasised.
Consider the outcome if you fail to take action X.	To ensure that the promised outcomes if you do X seems much better than if you don’t do X.  When there is a problem; it’s tempting to lunge toward any action because it makes you feel that you are taking some action rather than standing idle.
Consider the outcome if you fail to take action, this time using the inverse of Z as the outcomes	To ensure that doing something versus doing nothing looks just as appealing when you think about good outcomes as when you think about bad outcomes.  People are generally less willing to take risks to increase their gains – they would rather have a sure thing.  But they don’t want a sure thing for losses – they will take a risk to try to minimise them.
Evaluate whether the stripped promise is something you already know	To be sure that what’s being sold to you is something you can’t do yourself.  Technical talk – especially neuroscientific talk – can make old ideas seem cutting-edge.
Evaluate whether the change is clear; ‘clear’ means that you feel confident that you know how the Change will affect students’ minds	To ensure the Change is implemented as intended.  Changes that sound good can go awry if they are not implemented in the classroom as intended or if students don’t do what you’re hoping they will do.
Evaluate whether the outcome (Z) is clear; ‘clear’ means that there is a reasonably objective measure of whatever outcomes you expect, how big the change in the outcome will be, and when will it happen.	To be sure you will able to tell whether or not the promised outcome is happening
Check the outcome against the list of frequently claimed but unlikely-to-work promises	To be sure that claims are not made that are unfeasible, from a cognitive perspective – for example An improvement in all cognitive processes An improvement in a specific cognitive process (for example, critical thinking) irrespective of material An improvement for all students who struggle with a complex skill such as reading

Willingham goes onto argue that while considering the merits of a change it is necessary to write down your thoughts, and in doing so, this will lead so an increase in both the quantity and quality of reflection.  However, even if you have gone through the check-list and considered each of the items and have decided that the proposed change may be worth pursuing, there are still three further steps in the evaluative process, each of which will be the subject of a forthcoming post.

Implications for the School Research Leads

It is clear that School Research Leads are under constant time-pressure, with many colleagues being given only a small amount of time to undertake the role and using this check-list will take both time and effort.  However, given the increased and relentless resource pressures on schools over the next 5 years, anything that helps make sure time, money and effort are invested in the right change initiatives is worthy of consideration and use.   Furthermore, the role of the School Research Lead is still in its infancy, and if the role is to be continued to be supported and developed it is important that the role delivers discernible benefits.  The chances of delivering these benefits will be increased if  research-informed Changes are subject to rigorous pre-evaluation prior to implementation.  And in doing so, yo are more likely to deliver to changes which benefit your pupils, which is, at the end of day, what it is all about.

Reference
Willingham, D. (2012) When Can You Trust The Experts: How to tell good science from bad in education, Jossey-Bass, San Francisco.

The School Research Lead - Do we need to be vaccinated against the sharing of good practice virus?

The start of the academic year will see an outbreak of CPD and the sharing of good practice virus. Colleagues will spend hours,  if not days, being exposed to the self-espoused good practice of colleagues.  Unfortunately, like many viruses, the sharing of good practice 'virus' may do a great deal of harm, unless colleagues are vaccinated against infection from the well-meaning but unsubstantiated testimonials of colleagues.  Fortunately, a 'vaccine' has been developed by Daniel Willingham in his 2012 book - When Can You Trust the Experts : How to tell good science from bad in education.  Like all vaccines it can only provide the host with some protection against infection, as it will depend upon how the host responds to the vaccine.  However before we examine in more detail Willingham's vaccine, let's quickly remind ourselves of an activity we often see in a CPD and sharing of good practice day.

Sharing Good Practice - A Typical Activity

Some form of 'good-practice speed-dating' will no doubt form part of many schools start of term CPD events.  Colleagues will be asked to bring an example of what they perceive to be good practice and will be asked to explain how it works - often in less than two minutes -  to another colleague, with this process being repeated may be up to 20 - 30 times in a session. Unfortunately, there are a number of problems with this approach.  First, it gives insufficient time for colleagues to engage in an extended conversation about the 'good practice.  Willingham argues that such conversations are essential to ensure that the 'sharer of practice' fully understands his or her own practice and which is an important component of persuading others of the merit of a proposal.  However, the real problem with the speed-dating approach or others based upon testimonials is that it provides insufficient information to evaluate the practice being shared.  As Willingham states: You need not just stories of success from people who adopted the changes but also stories of failure.  You also need stories of success and failure from people who didn't adopt the change. (p 195)   The problem with testimonials in the sharing of good practice is illustrated in Figure 1

Figure 1 Information required to fairly evaluate a Change (Willingham, 2012 p 196)

	Things improved	Things didn’t improve
Adopted Change
Didn’t adopt Change (or adopted a placebo)

As such, Willingham argues that statements from individual teachers about his or her 'good-practice' only provides information from the top-left hand quadrant.  If we want to know whether some intervention or strategy has the potential to be good-practice than we will need information to allow to complete all four quadrants.

So what are we to do?

First it is essential for School Research Leads and CPD Co-ordinators to be aware of the design elements of effective professional development.  Fortunately, the Teacher Development Trust has recently provided us with a number of evidence-informed design elements, and about which I have written about previously.  Second, it is important that teachers talk about their practice in a 'disciplined' manner to determine whether espoused good practice is worth further discussion and possibly wider use within the school. Willingham suggests that when examining the potential of an educational change, strategy, or intervention teachers undertake a four-stage process.

1. Strip it - get rid of the fluff surrounding the idea/change/strategy/intervention and get to the heart of the actual claim.  What specific intervention, strategy or actions should the teacher adopt and what outcomes, say learning or achievement, are being promised.
2. Trace it - where did the idea come from.  Is the idea supported by a leading educational authority, as ironically in education this can be a weak indicator of validity or reliability 
3. Analyse it - what are you being asked to believe.  What is the evidence to support the claims being made?  How does this evidence relate to your own experience as a teacher?
4. Should I do it - is it something we already do.  Is it an old idea wrapped in new language? Has it failed previously in other settings with other students.  What are the opportunity costs 

'Flipping' Willingham to aid the sharing of practice 

 Willingham's four steps have been designed to help teachers discriminate between good and bad research evidence. Each of the four steps can be amended to help teachers structure the way they share practice with others.   In other words,  Willingham's four steps can be 'flipped' around for use by the teacher who is sharing the practice.  As such, if you intend to share practice with others the following elements needs to be covered.

• By doing X with my pupils there appears to be a Y percent chance that Z will happen
• Where the idea came from - is it supported by research evidence - if so, what are the criticisms of this research 
• State the 'class-based' evidence - what was measured, how was it measured, in comparison to what, how many pupils, how much did it help
• What were the opportunity costs associated with the change?  What negative-side effects were experienced? Who else was impacted upon?

Flipping the process around - with the steps being used by the sharer of practice has two main benefits.  It provides a structure for reflection by the teacher sharing the practice and which should facilitate more robust self-evaluation of practice before that practice is shared.   Second, it provides the 'audience' with a framework which allows them to 'test' through conversation how well the 'sharer'  understands his or her espoused good practice, which will in turn impact upon whether the sharees adopt the change.

In conclusion

As education can be a fashion-led profession and with fashions often based on so-called good practice it is essential that teachers can engage in the structured and disciplined evaluation of evidence.   Over the last 12 months as I have blogged about a evidence-informed practice,  attended events researchED events and followed twitter discussions, the need for teachers to be research literate is continually raised.  Unfortunately, there is very little discussion about what research literacy looks like in practice and how teachers should go about it.  With that in mind, if you are in the process of setting up a 'journal club' or are seeking a book to be read over the year, then Willingham's When Can You Trust the Experts : How to tell good science from bad in education is essential reading.  Future posts will help time-pressed School Research Lead by exploring Willingham's work in more detail, and expand upon each of the the four steps he recommends which will help you make significant progress to becoming a research literate and evidence-informed teacher.

Reference
Willingham, D. (2012) When Can You Trust The Experts: How to tell good science from bad in education, Jossey-Bass, San Francisco.