Lesson Study - The New Brain Gym?

If you are interested in teaching and learning, teacher professional development or school improvement, this post is for you.  Unfortunately, education is a 'fashion-led' profession and we are seeing  'fashion' now raise its head in the form of the widespread adoption of lesson study.  Lesson study is now being widely used as a mechanism to support the development teachers as inquirers.  Yet, the EEF's current view is There is some evidence that Lesson Study could have a positive impact in English schools, but not yet enough to justify scaling it up.  With this view position being  supported by Dylan Wiliam at the 2015 Sunday Times Festival of Education.

With that in mind, there appears to be little evidence of lesson study being critiqued or challenged as a way of supporting a teacher's learning before lesson study is implemented within an individual school.  Furthermore, in the reports published by schools who have used lesson study, there would appear to be little public reflection on either the lesson study process or examples of lesson study not meeting the needs of teacher development.  That does not mean that such reflection has not happened, it just does not appear to be in the public domain.

So what are we to do?   Given this lack of evidence, it would be sensible to look at alternative ways of supporting teachers develop their practice.  One approach is School Based Instructional Rounds (SBIR), which I will now examine in more detail.

Instructional Rounds (IR) :The background

Initially designed and developed by the Harvard Graduate School of Educations to be used within a network of schools, Instructional Rounds (IR) was inspired by the approach adopted by the medical rounds that doctors use as part of their training.   Although there are several versions of medical rounds, the essential elements involve doctors - be it trainees, interns, house officers, residents or consultants - observing and discussing the elements of a patient's diagnosis after a thorough review of the evidence and consideration of the various forms of intervention/treatment.    As Elmore et al (2009) state : The rounds model embodies a specific set of  ideas about how practitioners work together to solve common problems and to improve their practice.  In the education context, we call this practice instructional rounds, or rounds for short 





School Based Instructional Rounds : How does it work?

Although initially designed to work across a network of schools, the IR model has subsequently been developed to work at the level of individual schools.  So how does it work?

1. The school identifies a 'problem of practice' on which observers will focus during the classroom observation.
2. After a brief orientation, observers divide into groups to observer in three or four classrooms, spending about 20 minutes in each.
3. During the observations, observers write down specific non judgemental and non evaluative notes about what teachers and pupils are saying and doing in relation to the problem of practice.
4.  Following the observations, observers and participant teachers then analyse the data looking for patterns, and suggestions for improvements in practice or school systems (Adapted from Teitel, 2014)






What are the advantages of School Based Instructional Rounds?

SBIR have a number of potential advantages over network based IR, which Teitel identifies as follows:

• In-house teacher observers are likely to understand their context better than guest observers, which reduces the time necessary for briefings.
• In-house teacher observers have a more intimate knowledge of both pupils and what is being taught.  This is likely to give the observers a better handle on what is being learnt by pupils
• Given the more detailed nature of the observation and with it being conducted by colleagues, it is likely to lead to a speedier take up of actions required for improvement
• Follow-up - there is an embedded aspect of SBIR which makes it more likely that the outcomes of discussions will be followed-up and discussed.  As such, SBIR are far less likely to be seen as standalone events, with no follow-up.

Possible pitfalls 

Teitel goes onto identify two possile pitfalls of SBIR, and these include: one, teachers who know each other well staying in the 'land of nice;' two, teachers may not be able to notice what's happening as certain things are taken for granted.

School Based Instructional Rounds and School Improvement

Teitel identifies 5 key design questions which schools may want to consider, if they are going to incorporate SBIR into their school improvement work.

• Why develop SBIR?
• Who should be involved at the school, and why?
• What school-based meddles makes the most senses, and how will it work logistically?
• How will SBIR be integrated into existing improvement structures?
• How can educators take advantages of the benefits of SBIR and minimise the downsides.? (Teitel, 2014)

Some final words

Lesson study may have been adopted in your school after an extensive evidence-led evaluative process and works and that's great.  Lesson study may not have worked in your school and setting, and that's a shame for both colleagues and pupils.  Lesson study still requires further study to judge it whether it should be scaled-up.  With that lack of evidence in mind, SBIR offers an alternative to lesson study, which may or may not be right for you school, and that's a decision which is up to you.

Next week, I will be looking at another alternative to lesson study - Strategic Inquiry.

References

Elmore, R. F., Teitel, L., Fiarman, S. E., Lachman, A., & City, E. A. (2009). Instructional Rounds in Education: A Network Approach to Improving Teaching and Learning. Cambridge, MA: Harvard Education Publishing Group

Teitel, L. (2013)  School-Based Instructional Rounds: Improving Teaching and Learning Across Classrooms, Cambridge, MA: Harvard Education Publishing Group

Teitel, L. (2014) School-Based Instructional Rounds : Tackling problems of practice with teachers ; Harvard Education Letter, Vol 30 (1)

The School Research Lead - Can research help reduce teacher fallibility?

Sometimes on Twitter you read a tweet which cannot be answered by using the Uzi like rapid fire of 140 characters, and which deserves a far more considered response.  One such tweet,  recently came across my time-line

Now I had two initial reactions to this tweet.  One, was annoyance that whoever thought up the question for debate had not read my recent posts on when to trust the experts, so would already know the answer  to the question (research can suggest to teachers what to  look at, it cannot tell you whether it should be adopted, as that's the teacher's or school's decision).  More seriously, I was annoyed at the quality of the question as it was unlikely to lead to anything more than a bit of Sunday morning posturing and setting fire to straw-men.   For me, a far more relevant and useful question would be: can research help reduce teacher fallibility?  Now this might not be as 'sexy' or 'roll off the tongue' as well as 'Goodbye Mr Chips all that ..." but on the other hand, addressing the of question how research can reduce teacher fallibility is far more likely to be useful in helping teachers improve their day-to day-practice.  So to help me examine  the relationship between research and teacher fallibility, I have turned to Atul Gawande's 2010 book - The CheckList Manifesto - 

Gawande draws upon Samuel Gorowitz and Alasdair MacIntyre's 1976 essay : Toward a theory of medical fallibility - which sets out to understand why we fail.   Gorowitz and MacIntyre identify three reasons as to why we may fail to achieve what we initially set out to do:

Necessary fallibility

Some things we want to do are beyond our capacity.  Complex systems - such as schools - are beyond all-encompassing generalisations due to the differences in the circumstances of each individual school and complex feedback systems.  In these circumstances, the best possible judgement may turn out to be incorrect - even if it is based on the best available evidence and best-practice implementation.

Ignorance

The second source of fallibility is ignorance, we may not know enough of what works in teaching, learning and education.   There are still things we have yet to learn, examples include: how to make the most of peer-tutoring;  how best to integrate IT into teaching and learning; how best to develop teachers effectiveness in the latter years of their careers; what is best practice in setting and mixed ability teaching.  In other words, teachers will get things wrong because there are certain things we have yet to discover the first principles of getting it right.

Ineptitude

The final cause of fallibility is ineptitude, this is where the underpinning knowledge exists but we fail to apply it.   For example, we know that graded lesson observations lack validity and reliablility for accountability purposes but we still continue, in some cases, to carry them out.  We know that just providing a homework 'mark' is unlikely to lead to pupil learning.

As such, two kinds of teacher fallibility are inevitable; necessary fallibility, which derives from the nature of complex system and ability to understand them; and our current ignorance about how things works.   So what are we to do to reduce unnecessary ineptitude.

The humble checklist

Gawande thought the solution to reducing ineptitude could be found through the use of check-lists and  subsequently led a World Health Organisation study on the development of a Safe Surgery check-list.  A 19 point check-list was eventually developed - which included checks at three stages, pre-aneasthesia, pre-incision and at the end of the operation.  This checklist was subsequently piloted in eight hospitals around the globe, four of which were in high-income countries and four of which were in middle to low-income countries.  As a result of introducing the check-list there was 36 per cent reduction in serious complications post surgery, with deaths falling by 47 per cent.  Based on approximately 4000 patients, more than 150 patients were saved from harm, and 27 of them from death.

So the question becomes :  can checklists be used in educational context to improve teaching and subsequently pupil learning?  Now it could be argued that there is insufficient agreement about what constitutes high quality teaching and learning,  that even is where there is agreement that it is not possible to distil it down to a simple - pre, during and post lesson check-list.  However, even if this view is accepted, I would argue that checklists can help teachers codify their own personal knowledge and experience.  In other words, can teachers reduce ineptitude by using checklists of what they already know derived from their own personal experience.

Checklists and reducing teacher fallibility

The potential of this argument is reflected in Harry Fletcher- Wood's new book - Ticked Off: Checklists for teachers, students, school leaders - which is to be published in January 2016.

Fletcher- Wood argues that on many aspects of teaching, teachers already know what they would like to happen.  However, the day to day pressures associated with teaching means it is possible to forget what is already known, unless that learning is embedded within a series of habits.   Forgetting what we already know is likely to take place in stressful moments, where things have not worked out in the way we would like or unexpected events have taken place.   Fletcher-Wood argues that building in a pause point just before the start of a lesson is a way to ensure that all the things that needed to be done before a lesson, have been done.  Alternatively, teachers may wish to develop a quick check-list of things to consider before issuing a sanction to a student - and in doing so ensure that any sanction is both fair, equitable and proportionate.  In other words, checklists provide a way of doing things that we would do if we were not overtaken by the pressures of the moment.  And of course, checklists can be applied to more than just these two situations, they can cover a range of issues, such as having a difficult conversation with a member of staff.

So going back to our initial question of whether research can help reduce teacher fallibility, my argument is as follows.  Some forms of fallibility are inevitable/necessary given the complexity of school environment.  Some fallibility comes from ignorance, we just don’t know enough of how to make some things work.  On the other hand, some teacher fallibility is the product of ineptitude, that is not doing those things that we know that work.  Research has shown that checklists have the potential to reduce unnecessary ineptitude within medicine.  Even if one accepts that research cannot tell what or how to teach, checklists can help teachers make the most of what they already know and prevent missed opportunities for learning.

Note 

I'd like to thank Harry Fletcher-Wood for contributing to this blogpost

References

Fletcher-Wood, H. (forthcoming) Ticked Off: Checklists for teachers, students, school leaders, Crown House Publishing, London

Gawande, A (2010) The Checklist Manifesto : How to get things right, Profile Books, London

Gorowitz, S. and MacIntyre, A. (1976) Toward a Theory of Medical Fallibility : The Journal of Medicine and Philosophy, 1976, vol. I, no. I., p 51-71

The School Research Lead: Are you a positive deviant?

Are you - the School Research Lead - a positive deviant?  If not, you need to become a positive deviant yourself or at least help others to become one themselves.  The  underlying premise behind positive deviance is that in every organisations there are certain individuals whose uncommon practices/behaviours enable them to find better solutions to problems than their colleagues who have access to the same level of  resources.  So in schools, there are certain teachers whose uncommon teaching practices/behaviours enable them to find better solutions to problems than their colleagues who work in the same school and have access to the same resources.  So school research leads will need to develop evidence-informed behaviours and practices, which bring about improved pupil outcomes in comparison to other colleagues within the school.

In this post we will look at the work of Atul Gawande and his 2007 book - Better : A surgeon's notes on performance - and his recommendations on how to be a positive deviant.  Gawande is the author of a number of best-selling books, with his most recent being the The Checklist Manifesto and Being Mortal - Illness, Medicine, and What Matters in the End, and  is a surgeon in a Boston hospital, a Professor at the Harvard Medical School and a writer for the New Yorker.

So what does it look like to be a positive deviant?

Gawande has five suggestions for becoming a positive deviant?

1. Ask an unscripted question

On a day to day basis teaching involves hundreds of individual interactions with both pupils and colleagues.  Some of these interactions may be scripted, through the use of lesson plans, schemes of work or carefully planned meeting agendas.  With this in mind, one would hope that there would be plenty of time to ask unscripted questions to find out what's really going on for our pupils and colleagues.  Indeed, finding out what's really going for our pupils is an essential part of a number of processes - spirals of inquiry and strategic inquiry - and which should form part of the school research lead's repertoire

2. Don't complain

Throughout my own career in education, there were plenty of things I complained about.  I'm sure that many of you will agree with the me, that when teachers get together to talk about the profession - the tendency is to talk about what is not working, rather than what is.  However, as Gawande states : Resist it. It's boring, it doesn't solve anything, and it will get you down.  You don't have to be sunny about everything. Just be prepared with something else to discuss: an idea you read about, an interesting problem you cam across .... just keep the conversation going.

3. Write something

Gawande states : It makes no difference whether you write five paragraphs for a blog, a paper for a professional journal, or a poem for a reading group.  Just write.  Gawande argues that you should never underestimate the impact writing has both yourself and your world.  Gawande found thorough his own writing that he maintained his sense of purpose as medical practitioner, which may otherwise have been lost through the grind of day to day work.  He also argues by writing you make yourself part of a larger world.  Indeed, in the context of teaching and the emerging College of Teaching this may never have been so important.  From my own personal experience I would add - spend some time learning how to write better (and I know my own writing can be much improved) and read books such as On Writing Well by William Zinsser.

4. Change

Gawande argues that individual respond to change in one of three ways.  You have early adopters, if you have later adopters and there are some who are permanent sceptics and never adopt new ideas at all.  Gawande argues that you should make yourself an early adopter, although he is not arguing you should adopt every new fangled idea.  That said, you should be willing to reflect on your practice, identify what could be working better and find new ways of working.  As Gawande says : As successful as medicine  is, it remains replete with uncertainties and failure.  This is what make it human, at times painful, and also worthwhile. (p257) For me the same could be so easily said of education.


5. Count something

Count the number of students who find a particularly topic difficult.  Count the number of students who are having difficulty meeting homework deadlines.  Count the number of Y7 pupils who have had difficulties in English and maths, and which feeder school  they come from.  As Gawande states : If you count something interesting, you will learn something interesting (Gawande, 2007 p 255) So ignore David Didau's advice to resist, with all your might, the temptation to slap numbers on to your idea in an attempt to justify why it’s good; this is cargo cult science.

But education is not medicine, I hear you say.

And I agree,  education is not medicine, but that doesn't mean we cannot learn from medical practitioners and how they go about their work.  As is often said: a teacher from late 19th century would recognise today's classrooms, but would a 19th century surgeon recognise today's operating theatres. In other words, over the last 100 years which of the two professions has made the most progress.

Some final words

I'm going to end this post with Gawande's own closing words, to be a school research lead or evidence-informed teacher who is a positive deviant ...  find something new to try, something to change.  Count how often you succeed and how often you fail. Write about it.  Ask people what they think.  See if you can keep the conversation going (p257).

The School Research Lead - The Evidence Says Its A Good Idea - Should You Do It?

So you've read the research papers and separated the wheat from the chaff.  You've identified the research papers where it's just old ideas recycled with new terminology. Research papers where there are real doubts about the claims that are being made. Research papers where there just might be onto something but the evidence is incomplete and lacking.  This leaves research papers with changes which might just work, but you are not quite sure whether to adopt it or not.  Drawing once again upon the Daniel Willingham's 2012 book - When Can You Trust the Experts? How to tell good science from bad education - we will look at what can be done to help make a good and wise decision on whether to proceed with the change.  In doing so, we will consider the following:

• the factors to take into account of even if there is good evidence that the change is scientifically sound.
• a check-list to be completed before adopting a change 
• Willingham on education and evidence-based medicine.

Factors to take into account even if there is good evidence that the change is scientifically sound

Having read the research papers, you think you've come across an intervention/change for which there is sound evidence base and which you think might be worth adopting.  However, just because the evidence-base appears to be sound, that in itself is not a good enough reason to adopt the intervention/change.  There are a range of other factors which need to be taken into account before deciding whether or not to proceed.  Willingham identifies four factors which are worthy of consideration.

• Implementing a Change likely incurs a cost in times, energy and other resources.  Even if you believe that promised benefits will accrue you must weight them against the anticipated costs.
• Any Change you adopt will brings opportunity costs
• A Change may work as described but may have negative side effects
• Could the Change directly impact upon others -

A check-list to be completed before adopting a change

Willingham, drawing upon the work of Atul Gawande,  has developed a 10 point check-list which is to be completed before adopting a change.

1. The thing I'm hoping to change is ....
2. The way I can see that things change (in other words, what I'm going to measure) is ...
3. I've measured it before I start the Change , and the level it ....
4. I'm also going to measure ....... It probably won't be held by the Change, but you never know.
5. The Change could have some negative effects.  I'm most suspicious that it might influence ..... To be confident about whether or not it does, I'm going to measure ......
6. Here's how often I plan to collect the measurements, and the circumstances under which I'll do so: ......
7. My plan to keep these data organised is ....
8. The date by which I expect to see some benefit of the Change is ....
9. The size of the benefit I expect to see on that date ....
10. If I don't observe the expected benefit on that date, my plan is to ...... (Willingham, p 217)

At first glance, this may appear to be a daunting check-list.  On the other hand, as Woody Allen is often misquoted as saying -Eighty percent of success is showing up,  so by just asking these questions will get you most of the way towards success.

Willingham on education and evidence-based medicine

In writing a blog-post or any other publication it's quite easy to 'cherry-pick' and only quote 'authorities' when they agree with you.   In a number of blog posts I have taken the stance that education has much to learn from evidence-based medicine so it's important to note Willingham's reservations about making undue comparisons between education and medicine.  First, Willingham argues that in medicine there is a single goal shared by both doctors and patients i.e good health, whereas in education there are a far greater and diverse range of goals, aims and objectives.  Second, the deterministic model of science is relevant to medicine, well-evidence treatments for patients will probably work, whereas in education well-evidenced interventions may possibly work.  Willingham goes onto argue that architecture serves as a better comparison than medicine.  Architects have to manage multiple competing needs - be it functionality, form, the environment.  Furthermore, their decisions are informed by the 'science' of building, though not determined, in that they set the overall conditions for any design decisions.  As such, educational research should set the boundary decisions by which teachers make decisions about teaching and learning.   My own views on this matter, for what it is worth, is that whilst accepting the limitations in applying a strict interpretation of evidence-based medicine within education, there is a still a huge amount to be learnt from doctors, nurses and other health practitioners about how to make best use of the available evidence. For me, applying the processes associated with evidence-based medicine are likely to lead to you being a more  effective evidence-informed educational practitioner.

Some final words on this series of posts

This is the last of five posts which have been inspired by When Can You Trust The Experts: How to tell good science from bad in education.  As Daniel Willingham himself argues - the 4 steps outlined - will not make you into an expert in any particular area, as the 4 steps themselves are themselves a heuristic, a short-cut or work-around to help you get round that lack of expertise.  However, just because gaining expertise is a difficult and time consuming process, does not mean that you cannot be informed about the changes and interventions that have a reasonable chance of working for your students, and identifying those changes which are just BS and are not worthy of you and your pupils.

The School Research Lead - Getting Better at Analysing What the Experts Write and Say

In previous posts I've written about the need to 'strip, flip and trace' when trying to determine good educational science from the bad. In this post, and once again drawing upon the Daniel Willingham's 2012 book - When Can You Trust The Evidence: How to tell good science from bad in education - I will look at a number of steps which evidence-informed teachers and school research leads can take to effectively analyse educational research. The rest of this post will be in three sections:  

• what is the role of experience in analysing research; 
• Willingham's nine steps in analysing research; 
• the notion of practical significance.

The role of experience

The true test of friendship is not when you agree with someone, it's when you disagree them. I've got a huge amount of time and respect for Tom Bennett and for his work (along with Helen Galdin O'Shea) in developing the researchED movement. Unfortunately, in the following quote from his 2013 book - Teacher Proof - I think Tom gets it wrong.

… there are few things that educational science has brought to the classroom that could not already have been discerned by a competent teacher intent on teaching well after a few years of practice. If that sounds like a sad indictment of educational research, it is. I am astounded by the amount of research I come across that is either (a) demonstrably untrue or (b) patently obvious ... Here’s what I believe; this informs everything I have learned in teaching after a decade: Experience trumps theory every time. (Bennett 2013, 57-59)

Willingham argues that informal knowledge can mislead us in two ways; first, when we think with certainty; second, when we misremember or misinterpret past experience. As Willingham explains

... 'I know what happens in this sort of situation.' I think to myself, 'My daughter loves playing on the computer. She'll think reading program is great!' I might be right about my experiences - my daughter loves the computer - but that experiences happened to have been unusual; perhaps she loved the two programmes that she used, but further expertise will reveal that the she doesn't love to fool around with other programs. Another reason my experience might lead me astray is that I misremember or misinterpret my past experience, possibly due to confirmation bias. Perhaps it's not that my daughter loves playing on the computers; actually, I'm the one who loves playing on the computer. So I interpret her occasional, reluctant forays onto the Internet as enthusiasm. (p186).

So if teachers cannot relay on their experience to provide guidance on how to proceed then what are we to do?  Willingham helpfully identifies fours steps which can be taken to help manage our experiences;

• recognise that experience can be both fallible but it can also be insightful;
• check-out your experience with others. How does it relate to their experience or interpretations;
• think of the opposite to what your experience tells you. In other words, if you think of explanation or possible outcome, try and think of the exact opposite and see whether that is reasonable;
• actively look for daily examples of events which do not confirm past experience.

Willingham's Nine Steps Approach to Analysing Evidence

Having discussed the role of experience it is now appropriate to look in more details at the steps Willingham suggests you take to analyse evidence.  Before we do that it is necessary to define two terms - the Change and the Persuader.

The Change refers to a new curriculum or teaching strategy or software package or school restructuring plan - generically anything than someone is urging you to try as a way to better educate kids.

The Persuader refer(s) to any personal who is urging you to try the Change, whether it's a teacher, administrator, salesperson, or the President of the United States (Willingham, p136)

Willingham's nine steps to analyse evidence are summarised in Table 1.

Table 1 Actions to be taken when analysing evidence (Willingham, 2012 p 205)

Suggested Action	Why You Are Doing This?
Compare the Change’s predicted effects to your experience, but bear in mind whether the outcomes you’re thinking about are ambiguous and ask other people whether they have the same impression.	Your own accumulated experience may be valuable to you, but it is subject to misinterpretation and memory biases.
Evaluate whether or not the change Change could be considered a breakthrough.	If it seems revolutionary, it’s probably wrong.  Unheralded breakthroughs are exceedingly rare in science.
Imagine the opposite outcomes for the Change that you predict.	Sometimes when you imagine ways that an unexpected outcome could happen, it’s easier to see that your expectations was short-sighted It’s a way of counter-acting the confirmation bias.
Ensure that the evidence is not just a fancy labels.	We can be impressed by a technical sounding terms, but it may mean nothing more than an ordinary conversational term.
Ensure that bona fide evidence applies to the Change, not something related to the Change.	Good evidence for a phenomenon related to the Change will sometimes be cited as it proves the change.
Ignore testimonials.	The person believes that the Change worked, but he or she could easily be mistaken.  You can find someone to testify to just about anything?
Ask the Persuader for relevant research.	It’s a starting point to get research articles, and it’s useful to know whether the Persuader is aware of the research.
Look up research on the Internet.	The Persuader is not going to give you everything.
Evaluate what was measured, what was compared, how many kids were tested, and how much the Change helped.	The first two items get at how relevant the research really is to your interests.  The second two items get at how important the results are.

We know need to turn to the role of practical significance in determining how usefulness of research evidence for your practice.

Practical significance

In reading research articles you will come across the terms statistical significance and effect size.  Coe (2002) argues there is a difference between significance and statistical significance. Statistical significance means that you are justified in thinking that that the difference between the two groups is not just an accident of sampling. Effect size, on the other hand, is a way of measuring the extent of a difference between two groups (Higgins et al 2013). Now if we combine both effect size and statistical significance, this helps get a sense of the notion of practical significance of a change or intervention.

If the confidence interval includes zero, then the effect size would be considered not to have reached conventional statistical significance. The advantage of reporting effect size with a confidence interval is that it lets you judge the size of the effect first and then decide the meaning of conventional statistical significance. So a small study with an effect size of 0.8, but with a confidence interval which includes zero, might be more interesting educationally that a larger study with a negligible effect of 0.01, but which is statistically significant. (Higgins et al p6)

In other words, as Willingham states, practical significance refers to whether not that difference is something you care about (p 203). And, as such, requires you the reader to make a judgement call. Now making judgement calls about research evidence which you are not sure about is never easy. However, Willingham suggests three approaches to tackle the issue. 

• Make a mental note that you think the research maybe of practical significance. 
• If you have the opportunity raise the matter with the Persuader. 
• Ask how does the practical significance of the Change relate to your goals and what you are trying to achieve. Is the improvement which is being offered consistent with your objectives and the resources available to achieve them?

The next step

Having - flipped, stripped, traced and analysed the evidence - the next step is to consider whether the change should be adopted and that will be focus of a forthcoming post.

References

Coe, R. (2002) It's the Effect Size, Stupid What effect size is and why it is importantpaper presented at the Annual Conference of the British Educational Research Association, University of Exeter, England, 12-14 September 2002
Bennett, T. 2013. Teacher Proof: Why research in education doesn’t always mean what it claims, and what you can do about it. London: Routledge.
Higgins S., Katsipataki, M. Kokotsaki, Coe R., Elliot Major, L.  and  Coleman, R. (2013)  The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit: Technical Appendices
Willingham, D. (2012) When Can You Trust The Experts: How to tell good science from bad in education, Jossey-Bass, San Francisco.

researchED London 2015 and knowing when to trust the experts

In 19th century North America salesmen travelled across the continent promising wonder cures for all manners of ailments.  However, once the 'miracle-cure' was purchased and the salesman was onto his next town, his or her customers soon found they had been hoodwinked and the miracle-cure was nothing of the kind.  Snake-oil was one of the most commonly sold miracles cures and the term 'snake-oil' is still used today to describe so-called miracle-cures.  Indeed, in 21st century England, there are still the more than occasional sighting in schools of snake-oil salesmen, this time in the form of consultants, peddling the latest intervention which will transform pupil learning.  And despite Tom Bennett and Helen Galdin O'Shea's best efforts, one or two may have sneaked into researchED London 2015.  I really hope they haven't - but just in case they have - I've written this post.

So what are we to do?  Fortunately, Daniel Willingham  in his 2012 book,  When Can You Trust the Experts: How to tell good science from bad in education,  provides us with guidance in how to work-out  whether to believe what a so-called expert (consultant, ex-headteacher, educational researcher) says about a subject just because of his or her authority.  In particular, Willingham identifies a number of ways you can go wrong when you trust an authority.  But first, let's look at the structure of the argument as to whether claim is supported by 'scientific' method.

What happens when we believe an authority

Adapting Willingham (p170) -   say you - the researchED delegate  - don't understand the science behind an idea or claim made by an educational researcher/expert/researchEd speaker.  But you believe that the educational researcher/expert does.  You know that the educational researcher is saying the research evidence supports the idea or claim being made.  So you - the researchED delegate - without understanding the underpinning research, trust that the evidence supports the claim.  In other words, as the educational researcher/researchED speaker is an expert on the evidence, when the researchers says something about the evidence, you - the researchED delegate - are more likely to believe it.

However, there are a number of situations when this argument can go wrong - again adapted from Willingham.

• What we take to be signs of authority - acadamic pre-eminence, experience, twitter profile or educational columnist - turn out not to be very reliable, and the person is not, in fact, scientifically knowledgeable
• We might arrive at a false belief because we misundertand the position taken the academic, tweeter, consultant or columnist
• The 'authority' might be knowledgeable, but be in error takes up position on a topic outside his or her topic of expertise
• Two equally credible authorities might disagree on an issue, leaving it unclear what to do and how to proceed (adapted from Willingham p 178)

The School Research Lead's dilemma

This can be best summarised by F Scott Fitzgerald who said The test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function.  In this case the two opposing ideas are : I do not possess the expertise of an educational researcher, so I need to rely on those who do.  On the other hand, just because someone would appear to be a credible expert there are plenty of reasons not to believe such experts and authorities.  So what are we to do?  Unfortunately, and there's no other way round it, you the researchED delegate need to evaluate the strength of the evidence yourselves. This does not mean you have to become an instant expert in either qualitative or quantitative methods, but it does mean you have to have a list of questions which help unpack and understand educational research.  And in the case of researchED 2015 - make sure any speaker you listen to references a paper/book/article - where you can really unpick and analyse their argument.  If they haven't got a paper etc - then it's probably best to move onto someone who has.


Reference
Willingham, D. (2012) When Can You Trust The Experts: How to tell good science from bad in education, Jossey-Bass, San Francisco.

The School Research Lead and Stripping and Flipping - Essential skills for the evidence-informed practitioner

For teachers stripping and flipping may not sound like a way of helping them better do their jobs.  Indeed, it may even sound like the road to instant dismissal.  But seriously, stripping and flipping are two essential skills in becoming a research-literate and evidence-informed teacher.  Daniel Willingham in his 2012 book - When Can You Trust The Experts? How to tell good science from bad in education describes how stripping away verbiage and getting to the heart of actual claim being made, is the first step in a 'heuristic' process of evaluating educational interventions, programmes and research.  The rest of this post will now look at stripping and flipping in more detail. But first, lets define a few terms.

The Change refers to a new curriculum or teaching strategy or software package or school restructuring plan - generically anything than someone is urging you to try as a way to better educate kids.

The Persuader refer(s) to any personal who is urging you to try the Change, whether it's a teacher, administrator, salesperson, or the President of the United States (Willingham, p136) 

The Audience refers to individual teachers, department, schools, LEAs who are being asked to adopt the Change

Stripping 

In undertaking an evaluation of a Change Willingham states that it is essential to to be clear on the following:

1. precisely what Change is being suggested
2. precisely what outcome is being promised as a result of the Change
3. the probability that the promised outcome will actually happen if you undertake the Change (Willingham, p136)

For example : If an 'inadequate' school is converted to an academy, there is a 50 percent that the school will double the percentage of pupils gaining 5 GCSEs (inc English and Maths) at grade C or better

Flipping

Flipping is best explained by an example.  Let's suppose that a school claims that 65 percent of pupils leave school at 16 with a least a GCSE grade C in both English and Maths, then 35 percent of pupils leave school without at least a GCSE grade C in both English and Maths.  Now, all other things being equal, due to a framing effects a school with a 65 percent success rate is likely to be rated as better than a school with a 35 percent failure rate.

Things to watch out while flipping and stripping

Of course, there is far more to the stripping and flipping process than can be captured in a couple of examples.  Things to watch of for include"

• Emotional claims about the need or benefit of the Change
• Claims that the Persuader is like you
• False Analogies
• Postive and negative framing
• Old stuff which we have seen before
• Vague stuff which is difficult to discern either the output or outcomes
• Over-reach - claiming the Change will help all teachers and all pupils

Accordingly, there are a number of sub-steps to the Strip It/Flip It process and which are summarised in Table 1

Table 1 : Summary of the Strip It and Flip It Process

Suggested Action	Why You’re Doing This
Strip to the form “If I do X, then there is a Y percent chance of Z happening.”	To get rid of emotional appeals, peripheral cues, and proffered analogies that may influence your belief.  The scientific methods is suppose to be evidence-based and uninfluenced by these factors.
Consider whether the outcome (Z) has an inverse; if so, restate the stripped version of the claim using the inverse	To be sure that you appreciate all consequences of the action – for example, that an 85 percent pass rate implies a 15 percent failure rate.  We are subject to framing effects; we think something is better if the positive aspects are emphasised.
Consider the outcome if you fail to take action X.	To ensure that the promised outcomes if you do X seems much better than if you don’t do X.  When there is a problem; it’s tempting to lunge toward any action because it makes you feel that you are taking some action rather than standing idle.
Consider the outcome if you fail to take action, this time using the inverse of Z as the outcomes	To ensure that doing something versus doing nothing looks just as appealing when you think about good outcomes as when you think about bad outcomes.  People are generally less willing to take risks to increase their gains – they would rather have a sure thing.  But they don’t want a sure thing for losses – they will take a risk to try to minimise them.
Evaluate whether the stripped promise is something you already know	To be sure that what’s being sold to you is something you can’t do yourself.  Technical talk – especially neuroscientific talk – can make old ideas seem cutting-edge.
Evaluate whether the change is clear; ‘clear’ means that you feel confident that you know how the Change will affect students’ minds	To ensure the Change is implemented as intended.  Changes that sound good can go awry if they are not implemented in the classroom as intended or if students don’t do what you’re hoping they will do.
Evaluate whether the outcome (Z) is clear; ‘clear’ means that there is a reasonably objective measure of whatever outcomes you expect, how big the change in the outcome will be, and when will it happen.	To be sure you will able to tell whether or not the promised outcome is happening
Check the outcome against the list of frequently claimed but unlikely-to-work promises	To be sure that claims are not made that are unfeasible, from a cognitive perspective – for example An improvement in all cognitive processes An improvement in a specific cognitive process (for example, critical thinking) irrespective of material An improvement for all students who struggle with a complex skill such as reading

Willingham goes onto argue that while considering the merits of a change it is necessary to write down your thoughts, and in doing so, this will lead so an increase in both the quantity and quality of reflection.  However, even if you have gone through the check-list and considered each of the items and have decided that the proposed change may be worth pursuing, there are still three further steps in the evaluative process, each of which will be the subject of a forthcoming post.

Implications for the School Research Leads

It is clear that School Research Leads are under constant time-pressure, with many colleagues being given only a small amount of time to undertake the role and using this check-list will take both time and effort.  However, given the increased and relentless resource pressures on schools over the next 5 years, anything that helps make sure time, money and effort are invested in the right change initiatives is worthy of consideration and use.   Furthermore, the role of the School Research Lead is still in its infancy, and if the role is to be continued to be supported and developed it is important that the role delivers discernible benefits.  The chances of delivering these benefits will be increased if  research-informed Changes are subject to rigorous pre-evaluation prior to implementation.  And in doing so, yo are more likely to deliver to changes which benefit your pupils, which is, at the end of day, what it is all about.

Reference
Willingham, D. (2012) When Can You Trust The Experts: How to tell good science from bad in education, Jossey-Bass, San Francisco.