As an evidence-based
school leader one of the main challenges that you will face is knowing when and
whether you can trust expert advice. As the beginning of the academic year comes closer, you need to prepare yourself for the tidal wave of expert informed INSET/CPD which is about to
swamp schools and colleges. So knowing when to trust an
expert becomes particularly important.
However, not all experts agree, so what are you to do? Well, in the rest of this post we will look at an example of where experts disagree, and what strategies you can adopt when
faced with such a situation.
Dylan
Wiliam, Stephen Gorard and Significance Testing – Where experts disagree
Dylan Wiliam in his 2016 book : Leadership for Teacher Learning: Creating a
culture where all teachers improve so that all student succeed (Wiliam, 2016) writes extensively about
teachers and school leaders can learn from research. In a section on systematic reviews of
research – which includes a review of the challenges associated with the use of
randomised controlled trials in education – Wiliam states: Now there is no doubt that when RCTS produce statistically significant
results, they produce strong evidence for a causal relationship. (p75)
In this single sentence there is so much to
unpack and understand:
- What are RCTs?
- What do we mean by statistical significance?
- What is strong evidence?
- What do we mean by causal relationships?
Now
in this short blog post, it’s not possible to explore all the issues
associated with each of these terms. Instead, I’m
going to concentrate on just one issue – statistical significance – where there
would appear to be disagreement amongst the experts. And to help do this, I’m going to draw upon
the work of (Greenland et al., 2016) and (Gorard et al., 2017).
The very first problem that we face when
seeking to understand terms significance testing and p values is that as (Greenland et al., 2016) state: ‘... there are no interpretations of these concepts, which are at once simple,
intuitive, correct, and foolproof’ (p337)
This causes a real challenge for both the novice and expert researcher when trying to understand and apply to
concept of statistical significance, for as Greenland et al, go onto state
statistical significance is often misinterpreted
to imply that: ‘... statistical significance indicates
a scientifically or substantively important relationship had been detected’ (p341). Furthermore, they state that statistical
significance only indicates suggests that the data is unusual, but could also be of no real
interest.
The challenge of interpreting statistical
significance of the outcomes of randomised controlled trials is highlighted
further by (Gorard et al., 2017) who states: Statistical
significance just does not work – even when used as their advocates intended,
with fully randomised cases. They cannot be used to decide whether a finding is
worthy of further investigation or whether it should be acted upon on in
practice ( p28)
What
does this disagreement mean for you as an evidence-based school leader?
First, the debate and discussion over the
use and value of statistical significance testing live and controversial issue and is something you need to be aware of.
Second, even if you are not numerically minded
it’s probably worth spending a bit of time trying to understand the issues
associated with statistical significance. So have a look at general articles
such as https://aeon.co/essays/it-s-time-for-science-to-abandon-the-term-statistically-significant
and http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503
Third, whatever you may have learnt about the significance testing as an undergraduate or post-graduate, may
well be wrong.
Fourth, don’t assume just because there are
problems with statistical significance that randomised controlled trials have
little or no value – what matters is whether the research design is appropriate
for the research question (Gorard et al., 2017).
Fifth, unequivocal trust in anyone expert or groups of experts, is not an option.
And finally,
If you have any aspiration at all
of being an evidence-based school leader it requires a commitment to the time and
effort necessary to continually developing your research literacy – this is
not an event but a career long process.
Next week
We will look at range of questions that you can ask which will help judge the trustworthiness of quantitative educational research.
References
GORARD, S.,
SEE, B. & SIDDIQUI, N. 2017. The trials of evidence-based education.
London: Routledge.
GREENLAND, S., SENN, S., ROTHMAN, K., CARLIN, J.,
POOLE, C., GOODMAN, S. & ALTMAN, D. 2016. Statistical tests, P. European journal of epidemiology, 31, 337-350.
WILIAM, D. 2016. Leadership
for teacher learning, West Palm Beach, Learning Sciences International.
