NEW - Research schools and the limits of equipoise - are pupils at risk of unintended harm?

In last week's post I  began to discuss the ethical implication of schools not being clear in their understanding of the similarites and differences between research, quality improvement and evidence-based practice.  I argued that much of so-called 'research and evidence-informed practice' undertaken in schools was more akin to quality improvement than research.  However, ethical considerations are just as relevant to quality improvement activities as to research, and are often misunderstood and/or neglected.   In this post, I will argue:

• The current use of the notion of 'equipoise'  - we do not know for certain what works best - provides insufficient ethical protection for pupils - particularly when testing the withdrawal of an intervention.
• In these circumstances, it would be unethical to create a control group, where students were now denied access to a provision (such a 'research-design' would almost certainly not pass the scrutiny of a higher education provider's ethics committee)
• Schools need to give extremely careful consideration to the ethical considerations when conducting their own evaluations of interventions.
• More high quality guidance is required for schools in how to go thinking about through ethical considerations of interventions designed to bring about quality improvement
• Schools should develop partnerships with HEI to ensure appropriate ethical frameworks are in place.
• Though remember I'm just a blogger trying bridge the gap between research and practice, and as such my advice should be seen accordingly.

This may be particularly important to schools who are looking to withdraw existing provision due to being uncertain as to whether it works due to financial pressures.  What's more given all the Twitter chat about changes in marking policies, this post may be particularly timely.

What do we mean by equipoise ?

Probably the most authoritative guide to school-based evaluation of quality improvement intervenes is the EEF's DIY Evaluation Toolkit which provides some extremely useful advice on how to judge whether or not an intervention is effective.  The guide states

'Ethical evaluations start from a position of 'equipoise', where we do not know for certain what works best, it can be argued that that it is unethical not to try and establish which is more effective, particularly if the intervention is going to be repeated or rolled-out more widely. 

It is also often possible to evaluate interventions which avoid withholding them from pupils. For example, in many cases evaluation is about trying out different ways of delivering an intervention to see which works best. There is also a variety of other designs in which all participants get some form of intervention, described in more detail in the section below.

It is important to explain that an evaluation of a new possible method is taking place, not an evaluation of a better method as if we knew it was better, we’d be doing it already. In addition, it is a general principle of research that participants should give informed consent to take part. However, schools routinely innovate, try out new approaches and informally evaluate them all the time. You should use your own judgement and usual process when it comes to deciding whether to gain consent for children to take part in either the intervention or the testing.  (p9)

Now I can see how 'equipoise' can be used to justify the creation of a control group, when we don't know what works, as we are trying to make improvements in provision.  However, I'm not sure we can use the notion to justify the withdrawal of a provision from a control group.  Addressing this concern is particularly current given the pressure on both English schools' budgets and teacher workloads.

When equipoise is not enough?

A school may decide to withhold a particular intervention from some students - say additional subject support in mathematics prior to GCSE examinations - because they are unsure as to the intervention's cost-effectiveness.   So in this case, does equipoise provide sufficient ethical protection for pupils in the control group? Well, my first inclination was to say equipoise was unlikely to provide sufficient ethical protection as some pupils were likely to be disadvantaged by achieving a lower GCSE grade than would otherwise be the case.

So with that in mind, I contacted Professor Robert Slavin who is director of the Center for Research and Reform in Education at Johns Hopkins University, a part-time professor at the Institute for Effective Education at the University of York (England), and chairman of the Success for All Foundation.  Professor Slavin's response was extremely informative in that he states:

Every funded research project, and and most others, must pass ethical review. One standard component of ethical review is a determination that  the control group represents state of the art best practice, called "standard of care" in medicine. So if a school already provided tutoring or other special services and then withdrew them to form a control group, the study would not pass ethical review. Note that this has nothing to do with equipoise. (Slavin, 2016 private email correspondence with author)

In such circumstances, school wishing to explore the cost-effectiveness of additional support for GCSE mathematics, would need to compare:

....  the current program to an alternative that is equally or more likely to work and appealing to the participants.  It is not responsible to quit trying to improve, for example, GCSE performance, but it is both ethical and responsible to try alternative means of accomplishing that goal.

In other words, schools need to be extremely careful about the ethical implications of withdrawing an existing provision from some but not all pupils, if no appropriate alternative provision is put in place.

Some very tentative guidance

In many ways, I'm extremely reluctant to provide advice, as I'm just a blogger - and my posts rightly belong at the bottom of the pyramid of evidence.  On the other hand, I do think we need to start thinking through the ethical implications of schools engaging in research/quality improvement.  So with that in mind, Table 1 provides some very tentative guidance on how schools could proceed, depending on both the robustness of the available evidence and whether the intervention relates to new or existing practice.

Table 1 Guidance on Quality Improvement Initiatives

		Provision/Intervention
		New	Existing
Existing Evidence of what works, for who ….	Robust	Q1 Undertake small/large-scale pilot to identify local amendments required to make intervention work	Q2 Depending upon nature of evidence - enhance existing provision or withdraw provision entirely
	Inconclusive	Q3 Equipoise sufficient for RCT and use of control groups	Q4 Develop alternative equally or more likely to work, and at least as appealing (do not withdraw provision to create a control

In QI there is robust evidence of what works for who, in what context, for how long, to what extent, and how does it work.  It is decided to trial a new intervention as it is not clear what local amendments are needed to the intervention to make it work.  There may also be issues around the availability of resources and teacher commitment to the intervention. In Q2 there is robust evidence of works, or more importantly in this case what doesn't work.  In the former,  the provision should be enhanced as part of a process of continuous improvement, whereas in the latter it would be justified to completely withdraw the intervention.   In Q3 there is some evidence that an intervention may bring about improvement.  In such an instance, equipoise would provide justification for a randomised control trial, involving a control group.  In Q4, there is inconclusive evidence that an intervention works. In this case, if an evaluation is to be carried it is necessary to develop an alternative equally or more likely to work, and at least as appealing as the original intervention.  It would  not be ethical to withdraw the provision from some students to create a control group.  In addition, it is important that other ethical considerations, such as informed consent etc - also need to be taken into account to determine the ethical propriety of the intervention.

But to repeat - this is very, very tentative guidance and schools should look to work with a HEI partner to ensure research/quality improvement takes place within an appropriate ethical framework.

To conclude:

• The current use of the notion of 'equipoise'  - we do not know for certain what works best - provides insufficient ethical protection for pupils - particularly when testing the withdrawal of an intervention.
• In these circumstances, it would be unethical to create a control group, where students were now denied access to a provision (such a 'research-design' would almost certainly not pass the scrutiny of a higher education provider's ethics committee)
• In such circumstances it is necessary to develop an intervention which equally or more likely to work, as well as being appealing to the pupils.
• Schools need to give extremely careful consideration to the ethical considerations when conducting their own evaluations of interventions.
• More guidance is required for schools in how to go thinking through ethical considerations. 
• If in any kind of doubt, schools should contact a HEI to discuss the ethical implications of research/quality improvement activities

References

COE, R., KIME, S., NEVILL, C. & COLEMAN, R. 2013. THE DIY EVALUATION GUIDE.