Some useful definitions
- Causal claim - a claim in the form of "A was a cause of B".
- Causal role – how a change in A leads to a change in B
- Support factors – what factors need to be in place so that a change in A will lead to a change in B
Cartwright (2013) states that when arguing that ‘something works’ three quite different claims are often conflated
- The intervention works somewhere (there); the intervention causes the targeted effect in some individuals in some settings
- The intervention works: the intervention causes the target effect ‘widely’
- The intervention will work here: the intervention will cause the target effect in some individuals in this setting (adapted from Cartwright, p 98)
- The policy worked there (i.e. played a positive causal role in the causal principles that held there and the support factors necessary for it to play this positive role there were present for at least some individuals there).
- The policy can play the same causal role here as there
- The support factors necessary for the policy to play a positive causal role here are in place for at least some individuals here post-implementation p54
As such, Cartwright and Hardie argue that the effectiveness argument is like a three-legged stool – it doesn’t matter how strong one leg of the stool maybe, if either of the other legs fails – then the stool will fall over. Or put another way, it does not matter whether an intervention worked somewhere or causes the target effect widely – if either the causal mechanism or support factors are not in place – the intervention will not work here.
Cartwright and Hardie (2012) p55 and the three legged stool
What are the implications for you and your role as a school research lead?
The role of the school research lead/knowledge broker is not without significant challenges, difficulties and which also has a ‘dark-side’ - Kislov, Wilson, et al. (2016). So anything that can help you to be clear about the evidence and what it means, and which stops you over-claiming what the research tells you - should be useful.
Second, when colleagues come up to you and say ‘the research says intervention works – so let’s do it’ – it might be worth pausing and replying –‘that’s great news – so help me to be able to best help you – can you let me know more about – how the intervention worked and what was the context, within which it worked?’
Third, when reading research where there is little or no detail provided about the causal mechanism or support factors at work – then this research could be judged as being interesting but not necessarily that useful. Further reading into the what support factors are necessary for the intervention to succeed will be required.
This is the first of a series of posts which are going to cover ground which I hope is relevant to the start of the academic year. These posts will be on topics, such as; causal cakes; causal roles and mechanisms; what do we mean by evidence-based practice; and, when does evidence travel.
Cartwright, N. (2013). Knowing What We Are Talking About: Why Evidence Doesn't Always Travel. Evidence & Policy: A Journal of Research, Debate and Practice. 9. 1. 97-112.
Cartwright, N. and Hardie, J. (2012). Evidence-Based Policy: A Practical Guide to Doing It Better. Oxford. Oxford University Press.
Kislov, R., Wilson, P. and Boaden, R. (2016). The ‘Dark Side’of Knowledge Brokering. Journal of Health Services Research & Policy. 1355819616653981.
My new book due out October 2018 Evidence-Based School Leadership and Management: A practical guide