The school research lead and doing better evidence-based practice

One of the first successes of the Chartered College of Teaching has been to allow its members to access over 2000 educational journals, which had previously sat behind expensive paywalls.

Dame Alison Peacock, the college’s chief executive, said:

“Educational research can sometimes be seen as quite esoteric and separate from the realities of day-to-day teaching.

“We hope the knowledge and research platform we are developing will help to connect the big research ideas to classroom practice and allow our members to share their own insights on what works.”

However, having access to educational research in itself is not enough to guarantee for busy and time pressured teachers the usefulness of that research.  In this post, I will draw upon the work (Antonakis, 2017) who describes the conditions which are necessary for research to be useful.  I will then go onto to look at what Antonakis describes as five serious ‘diseases’ which hinder the production of useful research.  Finally, I will consider the implications of Antonakis’s analysis for your work as evidence-based teacher/school leader.

Three criteria for judging the usefulness of research

In Antonakis’s view the answers to three generic questions should provide some guidance as to usefulness of a research article and whether it may have an impact in your classroom or school.

• So what – does the research make a contribution to the research base – is it original or does it contribute to previous research efforts i.e does apply existing knowledge in a different context
• Is it rigorous – how accurate, robust and reliable is the research – does it accurately reflect the phenomena or causal process which has been uncovered?
• Will it make a difference to future research, or will the research inform either policy or practice?

So in other words, does the research tell us something new or add usefully to what we already know.  Is the research trustworthiness and been conducted to a high standard.  Has it got any use for either your current or future practice.

Five serious ‘diseases’ which stifle the production of useful research

Antonakis then goes on to claim that much published research is either not much used – i.e. has very few citations – or is not that useful, in that much published research is inaccurately reported by the authors.  In order to increase the usefulness of research Antonakis argues that science and social science need to tackle five ‘diseases’ which are acting impediments towards useful research.

Disease 1 - Significosis: an inordinate focus on the publication statistically significant results which leads to an overall biased distribution of estimates.  This bias in the distribution of estimates leads to a misrepresentation of what is actually ‘out there’ and impacts upon the distribution of effect sizes.  This then results in meta-analyses potentially overstating the impact of an intervention, as trials which did not produce statistically significant results are not published.  These meta-analyses then have the potential to misdirect both policy and practice.

Disease 2 – Neophilia: an excessive appreciation for newness and novelty.  Journals prefer to publish research findings that are new and novel, whereas much can be learnt from null results and attempts at replication.    Exploratory work is also required.

Disease 3 - Theorrhea: a mania for new theories which are very rarely subsequently tested.  Just as important to is identify compelling and interesting empirical relationships which can help with understanding

Disease 4 - Arigorium a deficiency in rigor of theoretical and empirical work.  In the context of education, if we were to make comparisons with subjects or disciplines there is relatively little or no agreement about models and methods.  For example, there is much discussion about distributed leadership or the role of trust in bring about school improvement, but little agreement about what these terms mean and how they can be empirically tested

Disease 5 - Disjunctivitis a tendency to produce large quantities of redundant, trivial or incoherent work. Researcher have an incentive to produce quantity so as to map out a reputations or develop a particular niche in a discipline.   This leads to the proliferation of short and speedy publications, which then adversely impacts upon the quality of systematic reviews.

The implications for you as an evidence-based teacher or school leader

Ultimately what is at stake here is the education of young people or the professional well-being of colleagues, so it is important to draw the right conclusions from research evidence.  So what can you as an evidence-based teacher/school do to vaccinate yourself from the five diseases that stifle useful research.

First, remember that research is just one of the four sources of evidence that are used by evidence-based practitioners.  As such, the so-called ‘prestige’ or ‘gold-standard’ associated with research, should not by default overpower claims and arguments supported by other sources of evidence.  In other words, just because research has been ‘published’ it does not by default mean that it is of any use.

Second, look out for systematic reviews which have made genuine attempts to incorporate non-published ‘grey’ research evidence which has been produced outside of the normal academic or commercial channels,  as these systematic reviews may give a better representation of the actual distribution of effect sizes.

Third, look out for research which is transparent with data, methods and reporting.  To the great credit of the Education Endowment Foundation they report all their research findings, even if the results have been disappointing, for example, the recent report on the work of school research champions

And finally

Remember when connecting research evidence to classroom practice – you need to develop a theory of action – and that’s another discussion.

Reference

ANTONAKIS, J. 2017. On doing better science: From thrill of discovery to policy implications. The Leadership Quarterly.