The school research lead and the evidence for coaching improving teaching and students' academic achievements

Recently a meta-analysis by (Kraft, Blazar, & Hogan, 2016) affirming the effectiveness of coaching as a developmental tool gained some prominence on twitter (see @GeoffreyPetty).  By aggregating results across 37 studies (Kraft et al, 2016) found pooled effect sizes of +0.57 standard deviations (SD) on instruction and +0.11 (SD) on achievement, which they claim affirms the effectiveness of coaching as a development tool.  Nevertheless, they recognised that in large-scale trials with more than 100 teachers involved – the effect size was half that found in small-scale studies.

Now the temptation is to say we that these findings justify an increased focus on coaching as a form of teacher professional development.  Indeed, it  might even be claimed that coaching is the 'magic bullet' of school improvement   However, with all these things, it’s never quite that simple.  So given that the ‘statistics’ in (Kraft et al., 2016) are ‘above my pay-grade’ I took my own advice (see School Research Leads need ‘mindlines’ not guidelines) and contacted @profstig (Professor Steve Higgins of the University of Durham) to see if the paper stood up to informed scrutiny.  So what did (Higgins, 2017) say?

One, it is a fairly robust meta-analysis. The techniques are sound and applied well.

Two, overall it looks like Bananarama again “the quality and focus of coaching may be more important than the actual number of contact hours” – the scale-up decrease also reinforces this message (0.17 to 0.08) for the drop in impact found from efficacy to effectiveness trials.

Three, “a one SD (i.e. ES = 1.0) change in instruction is associated with a 0.15 SD change in achievement” – looks pretty inefficient to me – this is a big change in behavior for what looks like a small gain in achievement. Some of the changes brought about by coaching must not be necessary or may be incorrect!

Four, the findings not likely to be the result of chance for reading (0.14 ** p<.01,), but within the margin for error for maths and science (smaller ES and fewer studies, so it was never going to reach statistical significance).

Five, when thinking about the overall benefit, for me, the issue is cost effectiveness. How expensive is coaching compared with one-to-one tuition per student? A lot cheaper I expect, so if you can get a reliable impact on student outcomes it is worth doing (particularly as the effects may continue for other groups of students if the teacher’s effectiveness has improved, whereas one-to-one benefits cease when you stop doing it).

Six, I don’t like their search strategy – I think it potentially builds in publication bias. I have no problem with expert informant recommendations, but then they needed to compensate for this selectivity by designing a systematic search which could find these studies (and the others which meet the same criteria. ‘Experts’ are likely to be pro-coaching and recommend (or remember) successful evidence. The trim and fill analysis suggests publication bias inflates the estimates (i.e. doubles it).

Finally – at least there is evidence of impact on student outcomes, meaning coaching can improve student outcomes!

Implications

So what are the implications of this research and discussion for you in your role as a school research lead.

To some extent, they depend upon your setting and context.  If you are a school research lead operating across a range of schools within a multi-academy trust (MAT) and where interventions are adopted across the whole (MAT) the results of any coaching intervention are likely to be significantly smaller than when first applied in a pilot school.  

Any intervention must be seen in terms of the ‘opportunity cost’ i.e. what would have been the value of the next highest valued alternative use of that resource – in terms of both resources and changes in teacher and pupil learning.   So it is important to think not only about the benefits but the long-term benefits and costs – and any negative unintended consequences – such as attention cost.

Regardless of your setting, it’s important not to be overwhelmed by the apparent complexity of research findings.  In this context, as long as you have an understanding of what is an effect size, and how big is an effect size, it’s possible to get some understanding of the issues.   So if we take a range of interpretations of the size of an effect size

• John Hattie and his estimate of average effect sizes being around +0.4.  
• The EEF’s DIY Evaluation Guide written by Rob Coe and Stuart Kime, where on pages 17 and 18 they provide some guidance on the interpretation of effect sizes (-0.01 to 0.18 low, 0.19 - 0.44 moderate, 0.45 - 0.69, high, 0.7 + very high) and with effect sizes being converted into months of progress. 
• Alternatively, if you are interested in the relationship between effect sizes and GCSE grades, you could turn to (Coe, 2002) where he note such, an improvement of one GCSE grade represents an effect size of about +0.5 – + 0.7. 
• (Slavin, 2016) who suggests an average effect size for a large study (250 participants) is 0.11.  

From this is should be quickly apparent that coaching is no more of a magic bullet than the average intervention, indeed it may have less than average effect, and it also may be a lot more expensive. 

Finally, take time to develop your ‘mindlines’ - (Gabbay & Le May, 2004)  i.e. collectively reinforced, internalised, tacit guidelines. These are informed by brief reading but also by your own and your colleagues’ experience, your interactions with each other and with opinion leaders, researchers, and other sources of largely tacit knowledge.   Modern technology and social media allow you to contact experts from outside of your own setting, most will be grateful that you have exhibited an interest in their work and more often that note hugely generous with both their time and expertise.

References

Coe, R. (2002). It's the effect size, stupid: What effect size is and why it is important.

Gabbay, J., & Le May, A. (2004). Evidence based guidelines or collectively constructed “mindlines?” Ethnographic study of knowledge management in primary care. Bmj, 329(7473), 1013. doi:10.1136/bmj.329.7473.1013

Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement: Routledge.

Higgins, S. (2017, 18 April, 2017). [Meta-analysis on coaching] personal correspondence

Kraft, M. A., Blazar, D., & Hogan, D. (2016). The Effect of Teacher Coaching on Instruction and Achievement: A Meta-Analysis of the Causal Evidence.

Slavin, R. (2016). What is a Large Effect Sixe.  Retrieved from http://www.huffingtonpost.com/robert-e-slavin/what-is-a-large-effect-si_b_9426372.html

The school research lead and the case against empathy

In a recent post, I argued that evidence-based practitioners need to display empathy if they are to act with what (Paul and Elder, 2014) describe as intellectual fair-mindedness.  Since I writing that post I have come across Paul Bloom’s recently published book Against Empathy: The case for rational compassion, in which Bloom argues that empathy can lead to poor decision-making.  As empathy can lead us to be be biased in favour of people we know or are like us - and biased us against the needs of others who are not like us, or we don’t know.   So with this in mind the rest of this post will examine the following:

• What do we mean by empathy and are there different types of empathy?
• How can some types of empathy lead to poor decision-making?
• What are the implications for school research leads and champions seeking to become conscientious, explicit and judicious evidence-based practitioners?

What do we mean by empathy and are there different types?

(Bloom, 2017) defines empathy as : … the act of coming to experience the world as you think someone else does (p16)

Bloom then goes onto cite Adam Smith who - using the term sympathy – describes empathy as the ability to think about another person and ‘place ourselves in his situation and become in some measure the same person as him, and thence form some idea of his sensations, and even feel something, which, though weaker in degree, is not altogether unlike them.

However, this form of empathy – known as emotional empathy -  is not the same as cognitive empathy, which is the … capacity to understand what’s going on in other people’s heads, to know what makes them tick, and what gives them joy and pail, what they see as humiliating and ennobling.p36  In other words, with cognitive empathy we are not talking about feeling your pain, but rather the ability to recognise that the other person is in pain.

So for the purpose of our discussion about the relationship between empathy, evidence-based practice and intellectual fair mindedness, we are interested in cognitive empathy and the ability to see someone else’s point of view, rather than trying to feel how they are feeling.  And it is cognitive empathy, which contributes to (Paul & Elder, 2014) and their notion of intellectual fair-mindedness.

How can emotional empathy get in the way of good decision-making?

So what is the case against emotional empathy.  Bloom argues Empathy is a spotlight focusing on certain people in the here and now.  This makes use care more about them, but it leaves us insensitive to the long-term consequences of our acts as well as to the suffering to those we do not or cannot empathise with.  Empathy is biased, pushing us in the directions of parochialism and racism.  It is short sighted, motivating actions that might makes things better in the short-term but lead to tragic results in the future.  It is innumerate, favouring the one over the many.  I can spark violence; our empathy for those close to use is a powerful force for war and atrocity toward others.  It is corrosive in personal relationships, it exhausts the spirit and can diminish the forces of kindness and love (p9)

Bloom backs up his claims with references to appropriate research, which I leave you to check out for yourself, as the purpose of this blog is not to provide the definitive empirical discussion about emotional empathy, but rather consider the implications for the evidence-based practitioner of emotional empathy.  For setting aside whether or not you agree with Bloom’s argument, a core tenant of being an evidence-based practitioner is to make explicit your assumptions and then explores the implications for being a conscientious, explicit and judicious evidence-based practitioner and school research lead.

What are the implications for school research leads and champions seeking to become conscientious, explicit and judicious evidence-based practitioners?

Given the claims Bloom has made about the downsides of empathy – be it whether empathy be a cause of war and racism – this is probably one of the most difficult ‘discussion’ sections I have written.   That said, I think we can draw out a number of implications which arise from Bloom’s analysis.

First, just because something seems intuitively that it is a good thing – does not necessarily mean that it is.

Second, given the distinction between cognitive and emotional empathy, when discussing research and evidence with colleagues, you need to be in command of your terminology.  Otherwise, discussions may become unfocussed as colleagues have different understandings of the terms being used.

Third, given that emotional empathy shines a ‘spotlight’ on the needs of certain individuals in the here and now. It may be that emotional empathy will steer your attention in the direction of current problems and the individuals who are experiencing them.  On the other hand, given the scarce time and resources available to the school research lead, it may be that there are other activities which may be more beneficial, particularly for pupils who have yet to join the school.

Fourth, in working with colleagues you are more likely to empathise with colleagues who are like you.  You may have been in teaching for say 5- 10 years and have vivid memories of what it is like to be an NQT, so you may cut NQTs or recently qualified colleagues some ‘slack’ in participating in knowledge-building activities.  On the other hand, you just might not ‘get’ the colleague who has been teaching for thirty years, and who has no apparent interest in research.  As a result, you are more willing to spend your time with the NQT rather than the more experienced colleague.

Fifth, there is a potential danger that when looking for problems of practice on which to focus your attention, you focus on groups of pupils who may have had the same problems or issues that you experienced when your age.  That said, there may be other pupils or colleagues for whom you have no ‘emotional’ connection, which means that their issues are potentially ignored.

And finally ....

Next week we will be looking at the evidence as to whether coaching leads to improved teaching and increases in pupil achievement.

  

References

Bloom, P. (2017). Against empathy: The case for rational compassion. London: Random House.

Paul, R., & Elder, L. (2014). Critical Thinking : Tool for retaking your professional life (Second Edition). New Jersey: Pearson.

The school research lead and the case for intellectual humility

In a recent post for the Chartered College of Teaching blog I argued that intellectual humility was a fundamental component of being a ‘conscientious, explicit and judicious’ evidence-based practitioner.    This is particularly important as for, as (Leary et al., 2016) note, much of what we believe to be true is either inaccurate, subject to bias or completely wrong.    So the rest of this post will draw upon the work of (Leary et al., 2016) to:  

• explain what is meant by ‘intellectual humility’; 
• report on research in which people recognised that their beliefs might be wrong; 
• identify questions for further research
• discuss the implications ‘intellectual humility’ for you in your role as school-research champion or research lead

Intellectual humility – what is it?

(Leary et al., 2016) define intellectual humility as recognizing that a particular personal belief may be fallible, accompanied by an appropriate attentiveness to limitations in the evidentiary basis of that belief and to one’s own limitation in obtaining and evaluating relevant information (p1)

As such, although intellectual humility is fundamentally about private assessments by an individual of the accuracy of their beliefs, intellectual humility is also manifested in how individuals respond to others, and which may or may lead to conflict.

Research on intellectual humility

(Leary et al., 2016) report on four research studies which have looked into the nature of intellectual humility.  One of the studies showed that intellectual humility was associated to variables related to openness, curiosity, tolerance of ambiguity, and low dogmatism.  In the other three studies, higher levels of intellectual humility were linked with people being: less certain about their religious beliefs; less inclined to see changes of opinions as flip-flopping; more attuned to the strength of persuasive arguments.

Questions for further research

(Leary et al., 2016) identify a number of questions which call for our attention.

• In what ways in individuals who possess high levels of intellectual humility compared to low levels of intellectual humility different in the way they process information?
• Do intellectual humble people possess more accurate and nuanced knowledge than those individuals who are less intellectually humble?
• How do different levels of intellectual humility influence individuals in how they handle differences of opinions or conflict with others.

Implications for you as a school research lead

There are a number of ways in which this discussion about ‘intellectual humility’ can inform your role as a school research champion.

First, there should be a degree of intellectual humility about the role of research and evidence in helping teachers bring about improvements in pupil outcomes.  We just don’t know whether providing teachers with access to research makes them better teachers, leading to improved outcomes for pupils

Second, given the diverse range of views within education about the pros and cons of different types of research and research methods, an appropriate stance to take would be that all methods have their uses, and may shed some light onto a problem of practice.  We probably don’t know enough to say we should prefer any one type of evidence or another.

Third, it’s important that as a school research lead, you demonstrate and role model intellectual humility, by displaying genuine curiosity and openness about the views of others.   In particular, it is important that you are not dogmatic and do not treat with disdain the views of colleagues who do not share your views about the value of research and other sources of evidence.

And finally ...

In my next post, I'll be looking at the relationship between empathy and being a 'conscientious, explicit and judicious' evidence-based practitioner.  The answer may well be surprising.

Reference

LEARY, M. R., DIEBELS, K. J., DAVISSON, E. K., JONGMAN-SERENO, K. P., ISHERWOOD, J. C., RAIMI, K. T., DEFFLER, S. A. & HOYLE, R. H. 2016. Cognitive and interpersonal features of intellectual humility. Personality and Social Psychology Bulletin, 0146167217697695.