The school research lead, the three legged stool and how to avoiding falling on your backside

As a school research lead you will have no doubt spent some of your summer reading research articles – be it systematic reviews or randomised controlled trials RCTS - and thinking about whether the interventions you’ve read about will work in your school and setting.  Indeed, you may have been working on a PowerPoint presentation making the case for why some well researched and evidenced teaching intervention which has shown positive outcomes in other schools, should be adopted within your school.  So to help with you with the task of developing an argument - which begins with ‘it worked there’ and which  concludes ‘it’ll work here’ -   I am going to lean on the work of Cartwright and Hardie (2012).  However, before I do that, it’s necessary to: define a number of terms, the understanding of which are central to getting out the most of out of this blogpost; then, clarify the causal claim being made when we say something ‘works’.

Some useful definitions

• Causal claim -  a claim in  the form  of "A was a cause of B". 
• Causal role – how a change in A leads to a change in B
• Support factors – what factors need to be in place so that a change in A will lead to a change in B

Clarifying what is meant by saying something works

Cartwright (2013) states that when arguing that ‘something works’ three quite different claims are often conflated

• The intervention works somewhere (there); the intervention causes the targeted effect in some individuals in some settings
• The intervention works: the intervention causes the target effect ‘widely’
• The intervention will work here: the intervention will cause the target effect in some individuals in this setting (adapted from Cartwright, p 98)

As such, evidence that supports a claims something works somewhere/there, is not evidence for saying that something will work here.  Cartwright and Hardie (2012) claim that what connects ‘it works somewhere’ with ‘it will work here’ is an argument that looks like this.

• The policy worked there (i.e. played a positive causal role in the causal principles that held there and the  support factors necessary for it to play this positive role there were present for at  least some individuals there).
• The policy can play the same causal role here as there
• The support factors necessary for the policy to play a positive causal role here are in place for at least some individuals here post-implementation p54 

In other words, if you want to show what worked (A) in another school or other schools in bringing about (B), you are going to need to show that the intervention can play the same causal role in your school – in other words, will a change in (A) trigger the same causal mechanism as it did in the other school or schools.  Second, you are going to have to show which of the support factors that were work, and which created the conditions for changes in (A) to bring about changes in (B) are in place and available in your setting.

As such, Cartwright and Hardie argue that the effectiveness argument is like a three-legged stool – it doesn’t matter how strong one leg of the stool maybe, if either of the other legs fails – then the stool will fall over.  Or put another way, it does not matter whether an intervention worked somewhere or causes the target effect widely – if either the causal mechanism or support factors are not in place – the intervention will not work here.

 Cartwright and Hardie (2012) p55 and the three legged stool

What are the implications for you and your role as a school research lead?

The role of the school research lead/knowledge broker is not without significant challenges,  difficulties and which also has a ‘dark-side’ -  Kislov, Wilson, et al. (2016).  So anything that can help you to be clear about the evidence and what it means, and which stops you over-claiming what the research tells you -  should be useful.

Second, when colleagues come up to you and say ‘the research says intervention works – so let’s do it’ – it might be worth pausing and replying –‘that’s great news – so help me to be able to best help you – can you let me know more about – how the intervention worked and what was the context, within which it worked?’

Third, when reading research where there is little or no detail provided about the causal mechanism or support factors at work – then this research could be judged as being interesting but not necessarily that useful.  Further reading into the what support factors are necessary for the intervention to succeed will be required.

What next 

This is the first of a series of posts which are going to cover ground which I hope is relevant to the start of the academic year.  These posts will be on topics, such as; causal cakes; causal roles and mechanisms;  what do we mean by evidence-based practice; and, when does evidence travel.

References

Cartwright, N. (2013). Knowing What We Are Talking About: Why Evidence Doesn't Always Travel. Evidence & Policy: A Journal of Research, Debate and Practice. 9. 1. 97-112.

Cartwright, N. and Hardie, J. (2012). Evidence-Based Policy: A Practical Guide to Doing It Better. Oxford. Oxford University Press.

Kislov, R., Wilson, P. and Boaden, R. (2016). The ‘Dark Side’of Knowledge Brokering. Journal of Health Services Research & Policy. 1355819616653981.

My new book due out October 2018 Evidence-Based School Leadership and Management: A practical guide

https://us.sagepub.com/en-us/nam/evidence-based-school-leadership-and-management/book257046