The school research lead : What works and the problem of side-effects

An essential element of the school research lead's role is to help incorporate evidence – research, school data, stakeholder views and practitioner expertise into the decision-making process.  Now to assist us with this task (Drucker, 2001) provides us with two useful ‘rules of thumb’ to be applied when incorporating the evidence into the decision-making process, which are:

• Act if on balance the benefits greatly outweigh cost and risk.
• Act or do not act, but do not ‘hedge’ or compromise.  (p196)

However, identifying the risks and costs of whether to adopt a new intervention or strategy may not be as straightforward as you might hope.   (Zhao, 2017) argues that educational research has largely ignored the potential harms arising from research into 'what works.'  As such, it may not be that easy for you as an evidence-based school research lead to identify the negative side effects associated with an intervention, which in turn makes it extremely difficult calculate whether the on balance the benefits of an intervention greatly outweigh the risks and costs.  So to aid with you with this task of weighing up costs and benefits, the rest of this post will consider the following:

• What do we mean by a side-effect
• Why do we need to study side effects
• What are the apparent reasons for the lack of concern for risks of interventions in education
• What are the implications for your role as a school research lead?

What do we mean by a side-effect?

Zhao uses the Cambridge Online Dictionary to define a side effect as an unwanted or unexpected results or condition that comes along with the desired effects of something.

Why do we need to study side-effects?

Well in the context of evidence-based school leadership and management the study of side-effects is extremely important.  For example, poorly selected, designed and implemented educational interventions may have significant detrimental impacts on the life changes of young people, or teacher’s reputations may be unnecessarily tarnished with failure.

Second, implicit within evidence-based practice is the notion of informed decision-making, with stakeholders in the decision being actively engaged in the decision-making processes, with their values, rights and preferences being taken into account.  This will be extremely difficult, if not impossible to do, unless stakeholders and decision-makers are fully aware of the side effects of an intervention.

Third, if we are going to have productive and useful discussions about ‘what works, for whom, to what extent, for how long, and in what context’ then it is necessary to research evidence which allows us to gain a fair minded view of the potential and costs of the intervention.  Zhao argues that research not including sufficient discussion of the side effects of an intervention leads to unhelpful discussions as to the pros and cons of an intervention, as opponents to the intervention may adopt extreme positions given the lack of consideration of side effects by the advocates of the intervention

What are the apparent reasons for the lack of concern for risks of interventions in education?

Zhao argues that there are probably many reasons as to the lack of concern about the risks associated with interventions.  First, given that education is perceived as a good thing, this leads to individuals potentially ignoring negative unintended consequences.  Second, the negative unintended consequences may take some time to emerge.  Third, if a narrow definition of success is adopted, then it becomes much more difficult to observe unintended consequences, as they may be deemed to be out of scope.  Finally, there may be a range of pressures – be it political, commercial or economic – which may lead to wilful blindness towards the side effects.

What are the implications for your role as a school research lead?

So if we accept (Zhao, 2017) argument about insufficient attention being paid by educational to the side-effects of interventions, what are the implications for your role as the school research lead in supporting evidence-based school leadership and management.

First, as (Kahneman et al., 2011) argues when making decisions we are prone to overconfidence, planning fallacies and optimistic biases.  As such, given the relative lack of research evidence on the side effects of intervention, you are going to have make sure there are attempts to identify potential side effects and negative unintended consequences before any decision is made.

Second, when appraising research evidence about an intervention a useful screen as to the trustworthiness of the evidence is whether there are statements and discussions about the side effects of the intervention.  If there is little or no discussion about side-effects then this should act as a warning sign that the research evidence is providing an incomplete report on what works.

Third, when receiving advice from external consultants or hearing the views of internal advocates of an intervention, it is extremely important to ask them about any anticipated side-effects of the intervention.  Again, depending on how well this questions is answered will determine the trustworthiness of the advice.

Fourth, if having decided to implement an intervention it is important to be vigilant for any side effects or unanticipated consequences.  However, it’s not enough just to focus on the subjects of the ‘intervention’ it is also important to consider the implications for ‘bystanders’ or other directly involved in the implementation of the intervention.  An intervention may provide benefits for its’ intended beneficiaries but the costs are being borne by others.

Fifth, to help identify the unintended consequences of an implementation then there is value in using the premortem technique which will help you imagine circumstances when your planned for intervention has failed.  By using the premortem technique this could reduce your chances of falling prey to both a range of cognitive biases and groupthink 

Sixth, just because an intervention has negative unintended consequences does not automatically mean that the intervention should be abandoned.  It may be that there are some low-cost mitigating strategies that can be adopted, which means you can still benefit from the intervention, but now at an acceptable level of unintended consequences.

And finally

If the anticipated results of an intervention are too good to be to true, then they probably are.  

References

DRUCKER, P. F. 2001. The Essential Drucker, Butterworth-Heinemann Oxford.

KAHNEMAN, D., LOVALLO, D. & SIBONY, O. 2011. Before you make that big decision. Harvard business review, 89, 50-60.

ZHAO, Y. 2017. What works may hurt: Side effects in education. Journal of Educational Change, 18, 1-19. which can be found at  http://zhaolearning.com/2017/02/17/what-works-can-hurt-side-effects-in-education/