When experts disagree?

As an evidence-based school leader one of the main challenges that you will face is knowing when and whether you can trust expert advice. As the beginning of the academic year comes closer, you need to prepare yourself for the tidal wave of expert informed  INSET/CPD  which is about to swamp schools and colleges. So knowing when to trust an expert becomes particularly important.  However, not all experts agree, so what are you to do?  Well, in the rest of this post we will look at an example of where experts disagree, and what strategies you can adopt when faced with such a situation.

Dylan Wiliam, Stephen Gorard and Significance Testing – Where experts disagree

Dylan Wiliam in his 2016 book : Leadership for Teacher Learning: Creating a culture where all teachers improve so that all student succeed (Wiliam, 2016)  writes extensively about teachers and school leaders can learn from research.  In a section on systematic reviews of research – which includes a review of the challenges associated with the use of randomised controlled trials in education – Wiliam states: Now there is no doubt that when RCTS produce statistically significant results, they produce strong evidence for a causal relationship. (p75)

In this single sentence there is so much to unpack and understand:

• What are RCTs?
• What do we mean by statistical significance?
• What is strong evidence?
• What do we mean by causal relationships? 

Now in this short blog post, it’s not possible to explore all the issues associated with each of these terms.  Instead, I’m going to concentrate on  just one issue – statistical significance – where there would appear to be disagreement amongst the experts.  And to help do this, I’m going to draw upon the work of  (Greenland et al., 2016) and (Gorard et al., 2017).

The very first problem that we face when seeking to understand terms significance testing and p values  is that as  (Greenland et al., 2016) state: ‘...  there are no interpretations of these concepts, which are at once simple, intuitive, correct, and foolproof’ (p337)

This causes a real challenge for both the novice and expert researcher when trying to understand and apply to concept of statistical significance, for as Greenland et al, go onto state statistical significance is often misinterpreted to imply that: ‘... statistical significance indicates a scientifically or substantively important relationship had been detected’ (p341).  Furthermore, they state that statistical significance only indicates suggests that the data is unusual, but could also be of no real interest. 

The challenge of interpreting statistical significance of the outcomes of randomised controlled trials is highlighted further by (Gorard et al., 2017) who states: Statistical significance just does not work – even when used as their advocates intended, with fully randomised cases. They cannot be used to decide whether a finding is worthy of further investigation or whether it should be acted upon on in practice ( p28)

What does this disagreement mean for you as an evidence-based school leader?

First, the debate and discussion over the use and value of statistical significance testing  live and controversial issue and is something you need to be aware of. 

Second, even if you are not numerically minded it’s probably worth spending a bit of time trying to understand the issues associated with statistical significance. So have a look at general articles such as https://aeon.co/essays/it-s-time-for-science-to-abandon-the-term-statistically-significant and http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503 

Third, whatever you may have learnt about the significance testing as an undergraduate or post-graduate, may well be wrong.  

Fourth, don’t assume just because there are problems with statistical significance that randomised controlled trials have little or no value – what matters is whether the research design is appropriate for the research question (Gorard et al., 2017).   

Fifth,  unequivocal trust in anyone expert or groups of experts, is not an option.

And finally,

If you have any aspiration at all of being an evidence-based school leader it requires a commitment to the time and effort necessary to continually developing your research literacy – this is not an event but a career long process. 

Next week

We will look at range of questions that you can ask which will help judge the trustworthiness of quantitative educational research.

References

GORARD, S., SEE, B. & SIDDIQUI, N. 2017. The trials of evidence-based education. London: Routledge.

GREENLAND, S., SENN, S., ROTHMAN, K., CARLIN, J., POOLE, C., GOODMAN, S. & ALTMAN, D. 2016. Statistical tests, P. European journal of epidemiology, 31, 337-350.

WILIAM, D. 2016. Leadership for teacher learning, West Palm Beach, Learning Sciences International.