Guest post - Meta-analysis: Magic or Reality, by Professor Adrian Simpson

Recently I had the good fortune to have an article published in the latest edition of the Chartered College of Teaching’s journal Impact in which I briefly discussed the merits and demerits of meta-analyses, Jones (2018).  In that article I lent heavily on the work of Adrian  Simpson (2017) who raises a number of technical arguments against the use of meta-analysis.   However, since then a blog post written by Kay, Higgins, and Vaughan (2018) has been published on the Evidence for Learning website, which seeks to address the issues raised in Simpson’s original article about the inherent problems associated with meta-analyses. In this post Adrian Simpson responds to the counter-arguments raised on the Evidence for Learning website.

Magic or reality: your choice, by Professor Adrian Simpson, Durham University

There are many comic collective nouns whose humour contains a grain of truth. My favourites include "a greed of lawyers", "a tun of brewers" and, appropriately here, "a disputation of academics". Disagreement is the lifeblood of academia and an essential component of intellectual advancement, even if that is annoying for those looking to academics for advice. 

Kay, Higgins and Vaughan (2018, hereafter KHV) recently published a blog post attempting to defend using effect size to compare the effectiveness of educational interventions, responding to critiques (Simpson, 2017; Lovell, 2018a). Some of KHV is easily dismissed as factually incorrect: for example, Gene Glass did not create effect size: Jacob Cohen wrote about it in the early 1960s; the toolkit methodology is not applied consistently: at least one strand [setting and streaming] is based only on results for low attainers while other strands are not similarly restricted (that is quite apart from most studies in the strand being about within-class grouping!)

However, this response to KHV is not about extending the chain of point and counter-point, but to ask that teachers and policy makers check arguments for themselves: Decisions about using precious educational resources needs to lie with you, not with feuding faculty. The faculty need to state their arguments as clearly as possible but readers need to check them: if I appeal to a simulation to illustrate the impact of range restriction on effect size (which I do in Simpson, 2017), can you repeat it - does it support the argument? If KHV claim the EEF Teaching and Learning toolkit use ‘padlock ratings’  to address the concern about comparing and combining effect sizes from studies with different control treatments, read the padlock rating criteria – do they discuss equal control treatments anywhere? Dig down and choose a few studies that underpin the Toolkit ratings – do the control groups in different studies have the same treatment?

So, in the remainder of this post, I invite you to test our arguments: are my analogies deceptive or helpful? Re-reading KHV’s post, do their points address the issues or are they spurious?

KHV’s definition of effect size shows it is a composite measure. The effectiveness of the intervention is one component, but so is the effectiveness of the control treatment, the spread of the sample of participants, the choice of measure etc. It is possible to use a composite measure as a proxy for one component factor, but only provided the ‘all other things equal’ assumption holds.

In the podcast I illustrated the ‘all other things equal’ assumption by analogy: when is the weight of a cat a proxy for its age? KHV didn’t like this, so I’ll use another: clearly the thickness of a plank of wood is a component of its cost, but when can the cost of a plank be a proxy for its thickness? I can reasonably conclude that one plank of wood is thicker than another plank on the basis of their relative cost only if all other components impinging on cost are equal (e.g. length, width, type of wood, timberyard’s pricing policy) and I can reasonably conclude that one timberyard on average produces thicker planks than another on the basis of relative average cost only if those other components are distributed equally at both timberyards. Without this strong assumption holding, drawing a conclusion about relative thickness on the basis of relative cost is a misleading category error.

In the same way, we can draw conclusions about relative effectiveness of interventions on the basis of relative effect size only with ‘all other things equal’; and we can compare average effect sizes as a proxy for comparing the average effectiveness of types of interventions only with ‘all other things equal’ in distribution.

So, when you are asked to conclude that one intervention is more effective than another because one study resulted in a larger effect size, check if ‘all other things equal’ holds (equal control treatment, equal spread of sample, equal measure and so on). If not, you should not draw the conclusion.

When the Teaching and Learning Toolkit invites you to draw the conclusion that the average intervention in one area is more effective than the average intervention in another because its average effect size is larger, check if ‘all other things equal’ holds for distributions of controls, samples and measures. If not, you should not draw the conclusion.

Don’t rely on disputatious dons: dig in to the detail of the studies and the meta-analyses. Does ‘feedback’ use proximal measures in the same proportion as ‘behavioural interventions’? Does ‘phonics’ use restricted ranges in the same proportion as ‘digital technologies’? Does ‘metacognition’ use the same measures as ‘parental engagement’? Is it true that the toolkit relies on ‘robust and appropriate comparison groups’, and would that anyway be enough to confirm the ‘all other things equal’ assumption?

KHV describe my work as ‘bad news’ because it destroys the magic of effect size. ‘Bad news’ may be a badge of honour to wear with the same ironic pride as decent journalists wear autocrats’ ‘fake news’ labels. However, I agree it can feel a little cruel to wipe away the enchantment of a magic show; one may think to oneself ‘isn’t it kinder to let them go on believing this is real, just for a little longer?’ However, educational policy making may be one of those times when we have to choose between rhetoric and reason, or between magic and reality. Check the arguments for yourself and make your own choice: are effect sizes a magical beginning of an evidence adventure, or a category error misdirecting teachers’ effort and resources?

  

References

Kay, J., Higgins, S. & Vaughan, T. (2018) The magic of meta-analysis, http://evidenceforlearning.org.au/news/the-magic-of-meta-analysis/ (accessed 28/5/2018)

Simpson, A. (2017). The misdirection of public policy: Comparing and combining standardised effect sizes. Journal of Education Policy, 32(4), 450-466.

Lovell, O. (2018a) ERRR #017. Adrian Simpson critiquing the meta-analysis, Education Research Reading Room Podcast, http://www.ollielovell.com/errr/adriansimpson/ (accessed 25/5/2018)

Evidence-informed practice and the dentist's waiting room

Sometimes the inspiration for a blogpost comes from an unexpected place, in this instance, my dentist’s waiting room.   Now I happen to be a regular visitor to my dentist because  back in 2005 I had a ‘myocardial infarction’ - better known as a heart-attack. Given that at the time I appeared to be fit, active and had completed many triathlons,  my heart-attack was ‘perplexing’ both for me and the medical professionals providing my treatment.  However, to cut a very long-story short, a contributory factor to my heart-attack appeared to be that  I had a bad-case of gum-disease and which research evidence suggests is  related to an increased risk of heart-disease,   Dhadse, P., Gattani, D., & Mishra, R. (2010).  And which is why I was in my dentist's waiting room about to  have my both teeth cleaned and gums ‘gouged’.

Now you may be asking, what on earth has an ‘evidence-based’  trip to do a dentist have to do with evidence-based or,  if you prefer, evidence-informed practice within schools.  Well it just so happened that whilst in the dentist’s waiting room I was reading Hans Rosling’s recently published book: Factfulness: Ten reason we’re wrong about the world – and why things are better than you think, when I came across this  paragraph about mistrust, fear and the inability to ‘hear data-driven arguments.

 In a devastating of example critical thinking gone bad, highly educated, deeply caring parents avoid the vaccinations that would protect their children from killer diseases.  I love critical thinking and I admire scepticism, but only in a framework that respects evidence.  So if you are sceptical about the measles vaccinations, I ask you to do two things.  First, make sure you know what it looks like when a child dies of measles.  Most children who catch measles recover, but there is still no cure and even with the best modern medicine, one or two in every thousand will die from it.  Second, ask yourself, “What kind of evidence would convince me change my mind about vaccination.  If the answer is ‘no evidence could ever change my mind about vaccination,” then you are putting yourself outside evidence-based rationality, outside the very critical thinking that first brought you to this point.  In that case, to be consistent in your scepticism about science, next time you have an operation please ask your surgeon not to bother washing her hands.  (p117).

So what are the implications Rosling et al’s critique of critical thinking gone wrong  for your role a school leader wishing to promote the use of evidence within your school.   At first glance, it seems to me that there are three implications.

First, ask yourself the question for about an issue which have pretty strong views – be it mixed-ability teaching, grammar schools and the 11 plus, or progressive vs traditional education – “What evidence would it take to change your mind?”  This is important as a critical element of being a conscientious evidence-informed practitioner is to actively seek alternative perspectives.  And if you are not at least willing to be persuaded by those perspectives, there is little point seeking  them out in the first place

Second, when working with colleagues who may ‘reject’ evidence-informed practice – ask them the same question “What evidence would it take to change your mind?”.  If they respond “there is no evidence that would get me to change my mind” ask them the following question: “Ok, is there a teaching approach you particularly favour, and if so, why?” and then ask the follow-question – “Tell me more.”

Third, there may be occasions when working with colleagues are resistant to evidence-informed practice that you have to resort to a variant of the  ‘surgeon with dirty hands’ argument, so ask the following: “Would you like your own children or children of family members to be taught by a teacher or teachers who: 

• Do not have a deep knowledge and understanding of the subjects they teach
• Have little or no understanding about how pupils’ think about the subject they are teaching
• Are not very good at asking questions
• Do not review previous learning
• Fail to provide model answers
• Give adequate time for practice for pupils to embed their skills
• Introduce topics in a random manner
• Have poor relationships with their pupils
• Have low expectations of their pupils
• Do not value effort and resilience
• Cannot manage pupil behaviour
• Do not have clear rules and expectations
• Makes inefficient and ineffective use of time in lessons
• Are not very clear in what they are trying to achieve with pupils
• Haven’t really thought about how learning happens and develops or how teaching can contribute to it.
• Give little or no time to reflecting on their professional practice
• Provide little or no support for colleagues
• Are not interested in liaising with pupils’ parents
• Do not engage in professional development  (amended from Coe, Aloisi, et al. (2014)

And if they answer No – we would not want my children or family members taught by such teachers – then you might respond by saying “You might not believe in evidence-informed practice though you would appear to agree with the evidence on ineffective teaching.”

And finally

Working with colleagues who have different views to you on the role of evidence-informed practice is inevitable.  What matters is not that you have different views but rather how do you about finding the areas you can agree on, which then gives you something to work on in future conversations

References

Coe, R., Aloisi, C., Higgins, S. and Major, L. E. (2014). What Makes Great Teaching? Review of the Underpinning Research. London.

Dhadse, Prasad, Deepti Gattani, and Rohit Mishra. “The Link between Periodontal Disease and Cardiovascular Disease: How Far We Have Come in Last Two Decades ?” Journal of Indian Society of Periodontology 14.3 (2010): 148–154. PMC. Web. 29 May 2018 

Rosling, H, Rosling, O., and Rosling Ronnlund, A. (2018).  Factfulness: Ten reason we’re wrong about the world – and why things are better than you think, London: Sceptre

Guest Post : Unleashing Great Teaching by David Weston and Bridget Clay

This week's post is a contribution from David Weston and Bridget Clay who are the authors of Unleashing Great Teaching: the secrets to the most effective teacher development, published May 2018 by Routledge. David (@informed_edu) is CEO of the Teacher Development Trust and former Chair of the Department for Education (England) CPD Expert Group. Bridget (@bridge89ec) is Head of Programme for Leading Together at Teach First and formerly Director of School Programmes at the Teacher Development Trust.

Unleashing Great Teaching 

What if we were to put as much effort into developing teachers as we did into developing students? How do we find a way to put the collective expertise of our profession at every teacher’s fingertips? Why can’t we make every school a place where teachers thrive and students succeed? Well, we can, and we wrote Unleashing Great Teaching: the secrets to the most effective teacher development to try and share what we’ve discovered in five years of working with schools to make it happen.  Quality professional learning needs quality ideas underpinning it. But, by default, we are anything but logical in the way that we select the ideas that we use. A number of psychological biases and challenges cause us to reject the unfamiliar.

We all have existing mental models which we use to explain and predict. To challenge one of these models implies that much of what we have thought and done will have been wrong. We all need to guard against this in case it leads us to reject new ideas and approaches. This is nothing new.

In 1846 a young doctor, Ignaz Semmelweis, suspected that the cause of 16% infant mortality in one clinic might be the failure of doctors to wash their hands. When he ran an experiment and insisted that doctors wash hands between each patient, the deaths from fever plummeted. However, his finding ran so against the established practice and norms that his findings were not only rejected but widely mocked despite being obviously valid. This reactionary short-sightedness gave rise to the term The Semmelweis Reflex: ‘the reflex-like tendency to reject new evidence or new knowledge because it contradicts established norms, beliefs or paradigms.’

An idea that contracts what we already think, which comes from a source that we don’t feel aligned to, or which makes us feel uneasy, is highly likely to be rejected for a whole range or reasons , even if there is a huge amount of evidence that it is far better than our current approach.

Reasons for rejection

Confirmation bias is really a description of how our brains work. When we encounter new ideas, we can only make sense of them based on what’s already in our heads, adding or amending existing thinking. This means that anything we encounter that is totally unfamiliar is less likely to stick than something partially familiar. Similarly, an idea that is mostly aligned with our existing thinking is more likely to stick than something completely at odds – the latter is a bit like a weirdly-shaped puzzle-piece: it’s very hard to find a place for it to go.  The effect of all of this is that when we hear an explanation, we remember the ideas that confirm or support our existing thinking and tend to reject or forget the ideas that don’t.

But it’s not just the nature of the ideas that affect our ability to learn. If an existing idea is associated with the memory of lots of effort and hard work, it becomes harder to change. This sunk cost bias means that we excessively value things we’ve worked hard on, no matter whether they’re actually very good or not. This bias is also known as the Ikea Effect – everyone is rather more proud of their flatpack furniture than this cheap and ubiquitous item perhaps deserves, owing to the effort (and anger!) that went into its construction.

We also see a number of social effects that mean that we don’t just listen to other people’s ideas in a neutral way. The Halo Effect is the way we tend to want to believe ideas from people we like and discount ideas from people we don’t. Of course, none of that bears any relation to whether the ideas are good. Public speakers smile a lot and make us laugh in order to make the audience feel good and thus become more likely to believe them. Two politicians of different parties can suggest the exact same idea, but supporters of the red party are much more likely to hate the idea if they hear it from the blue politician, and vice versa. A teacher from a very different type of school is much less likely to be believed than someone you can relate to more – though of course none of this necessarily affects whether their ideas are good.

If someone does present an idea that conflicts with our current thinking and beliefs, they run the risk of Fundamental Attribution Error. When we come into conflict with others, we rush to assume that the other person is of bad character. Any driver who cuts you up is assumed to be a terrible driver and a selfish person, but if you cut someone else up and they hoot then you generally get annoyed with them for not letting you in. A speaker or teacher who tells you something you don’t like is easily dismissed as ignorant, annoying or patronising.

Using evidence to support professional learning 

So how do we ensure that we’re using quality ideas to underpin professional learning? In our book we lay out some tools to help you overcome your inevitable biases.

Firstly, it’s very useful to look out for systematic reviews. These are research papers where academics have carefully scoured all of the world’s literature for anything relevant to a topic, then categorised it by type, size and quality of study, putting more weight on findings from bigger, higher quality studies and less on smaller, poorly-executed research. They bring all of the ideas together, summarising what we appear to know with confidence, what is more tentative, and where there are areas where the evidence is conflicting or simply lacking.

If you are interested in a topic, such as ‘behaviour management’ or ‘reading instruction’ then it’s a really good idea to tap it into a search engine and add the words ‘systematic review’. Look for any reviews conducted in this area to get a much more balanced view of what is known.

Secondly, raise a big red flag when you can feel yourself getting excited and enthusiastic about an idea. That’s your cue to be extra careful about confirmation bias and to actively seek out opposing views. It’s a very helpful idea to take any new idea and tap it into a search engine with the word ‘criticism’ after – e.g. ‘reading recovery criticism’ or ‘knowledge curriculum criticism’.

Thirdly, be a little more cautious when people cite lists of single studies to prove their point. You don’t know what studies they’ve left out or why they’ve only chosen these. Perhaps there are lots of other studies with a different conclusion – only a good systematic review can find this out.

Finally, be cautious of the single study enthusiasm, where newspapers or blogger get over-excited about one single new study which they claim changes everything. It may well be confirmation bias – or indeed if they are criticising it then it could also be confirmation bias causing them to do so.

To conclude

Of course, good quality ideas are only one ingredient. In our book we also explore the design of the professional learning process, offer a new framework to think about the outcomes you need in order to assist in evaluation, and discuss the leadership, culture and processes needed to bring the whole thing together. There are many moving parts, but if schools can pay the same attention to teachers’ learning as they do to students’ learning, we can truly transform our schools and unleash the best in teachers.

.