Friday, 17 August 2018

The school research lead and RCTs - what can a systematic review tell us?

As a school research lead one of the things that you will have to get grips with is the debate over whether randomised controlled trials (RCTs) can make any meaningful contribution to understanding ‘what works’ in educational settings.  Helpfully  Connolly, Keenan, et al. (2018)  have recently had published systematic review on the use of RCTs in education, which seeks to address four key criticisms of RCTs: it is not possible to undertake RCTs in education; RCTs are blunt research designs that ignore context and experience; RCTs tend to generate simplistic universal laws of ‘cause and effect’; and that they are inherently descriptive and contribute little to theory. So in this rest of this post I will provide extracts from the systematic review, examine the review’s answer to the questions posed, identify some missed opportunities, and finally, make some comments about RCTS and the work of school research leads.

Connolly, et al. (2018) systematic review - extracts

The systematic review found a total of 1017 unique RCTs that have been completed and reported between 1980 and 2016, with three quarters of these being produced over the last 10 years.

Over half of all RCTs identified were conducted in North America and a little under a third in Europe. 

The RCTs cover a wide range of educational settings and focus on an equally wide range of educational interventions and outcomes. 

Connolly et al go onto argue that the review: provides clear evidence to counter the claim that it is just not possible to do RCTs in education. As has been demonstrated, there now exist over 1000 RCTs that have been successfully completed and reported across a wide range of educational settings and focusing on an equally wide range of interventions and outcomes. Whilst there is a clear dominance of RCTs from the United States and Canada, there are significant numbers conducted across Europe and many other parts of the world. Many of these have been relatively large-scale trials, with nearly a quarter (248 RCTs in total) involving over one thousand participants. Moreover, a significant majority of the RCTs identified (80.8%) were able to generate evidence of the effects of the educational interventions under investigation.

As noted earlier, these  figures are likely to be under-estimates given the limitation of the present systematic review, with its restricted focus on articles and reports published in English. Nevertheless, the evidence is compelling that it is quite possible to undertake RCTs in educational settings. Indeed, across the 1017 RCTs identified through this systematic review, there are almost 1.3 million people that have participated in an RCT within an education setting between 1980 and 2016 ……

Secondly, there is some evidence to counter the criticism that RCTs ignore context and experience. Whilst they only constitute a minority of the trials identified (37.7%), there were 381 RCTs found that included a process evaluation component…..

Thirdly, there is more evidence to suggest that the RCTs produced within the time period
have attempted to avoid the generation of universal laws of ‘ cause and effect’ . Certainly, those RCTs identified that have included at least some subgroup analyses would suggest a more nuanced approach amongst those conducting RCTs, that acknowledges that educational interventions are not likely to have the same effect across all contexts and all groups of students. Moreover, this is clearly evident amongst the majority of RCTs reported (77.9%) that included at least some discussion of and reflections on the limitations of the findings in terms of their generalizability….

…. (in) relation to the fourth criticism regarding the a theoretical nature of RCTs, this is also challenged to some extent by the findings presented above. A clear majority of RCTs that were reported included some discussion of the theory underpinning the interventions under investigation (77.3%). Moreover, a majority of RCTs (60.5%) also provided some reflections on the implications of their findings for theory….

Overall, the findings from this systematic review of RCTs undertaken in education 1980 – 2016 are mixed. On the one hand, there is clear evidence that it is possible to conduct RCTs in education, regardless of the nature of the education setting or of the particular type and focus of the intervention under consideration….

On the other hand, it is perhaps not surprising that criticisms of RCTs continue when nearly two thirds of RCTs in this period of time have not included a process evaluation component and where nearly half of them have not looked beyond the overall effects of the intervention in question for the sample as a whole. Similarly, it is difficult to challenge the view that RCTs promote a simplistic and atheoretical approach to educational research when nearly 40% of trials in this analysis have failed to reflect upon the implications of their findings for theory.

Have the criticisms of RCTs been answered?

The main assumption of the systematic review is that it is possible to confirm whether RCTS can be carried out in education by merely counting the number of RCTs – subject to certain criteria – have been carried out.  However, this does not give you an answer as to the question as to whether it’s possible to carry out RCTS in an educational context.  It merely tells you, that a number of RCTS have been carried out by researchers and commissioners of research who believe it is possible to carry out RCTS within education.  

Missed opportunities

One of the first things that struck me about the systematic review was that it appeared to give very little attention, if any, as to whether any of the RCTs had systematic flaws in their design.  It may well be that over a 1000 RCTs have been conducted between 1980 and 2016 but a recent review of RCTs by Ginsburg and Smith (2016) reported that 27 RCTs that met the minimum standards of the What Works Clearinghouse based in the US, found that 26 of the RCTS had serious threats to their usefulness.  This would suggest that may be only 40 or so of the RCTS included in Connolly’s review did not have some kind of the serious threat to their trustworthiness.  Second, we then need to look at how many of these 40 or so RCTs included a process evaluation, made a contribution to theory and did not seek to overgeneralise.  It’s likely to a be very small percentage of the number RCTS which had been carried out. In other words, out of 1000 plus RCTs how many did not have fundamental flaws in design or some other failing.

What does this all mean for you as a school research lead?

For a start, Connolly et al provide an accessible introduction to a discussion of the issues associated with RCTs.  As such, it’s worth spending time reading the review.

Second, the review highlights some of the issues associated with the hierarchies of evidence.  Both systematic reviews and RCTS appear to be near the top of hierarchies of evidence.  However, what matters more is whether the research design is applicable to the research question at hand, Gorard, See, et al. (2017) and Sharples (2017).

Third, when reading RCTs it is necessary to have some kind of check-list so that you judge the trustworthiness of what you are reading.  In my forthcoming book, Evidence-Based School Leadership: A practical guide,  Jones (2018) - I try and do this by combing the work of Gorard, et al. (2017) and Ginsburg and Smith (2016).  For example, look out for whether the intervention is being evaluated by the researchers who developed the intervention.

Fourth, setting aside the issue of whether it’s possible to conduct an RCT within education, assuming that it is, theirs is the issue about how the analysis of the results is carried out, and whether p-values and statistical significance have been misused.  This is not the time or place to go into detail about this debate – that said I would recommend you have a look at Wasserstein and Lazar (2016) – and the American Statistical Association’s statement on p-values - Gorard, et al. (2017) with more adventurous readers having a look at Greenland, Senn, et al. (2016).  If on the other hand, you fancy having a look at something which is far more accessible, I recommend you have a look at Richard Selfridge’s new book – Databusting for Schools.

And finally

RCTS within educational research are not going away any time in the near future and I hope this post has provided a little more clarity

References

Connolly, P., Keenan, C. and Urbanska, K. (2018). The Trials of Evidence-Based Practice in Education: A Systematic Review of Randomised Controlled Trials in Education Research 1980–2016. Educational Research.
Ginsburg, A. and Smith, M. (2016). Do Randomized Controlled Trials Meet the “Gold Standard”? American Enterprise Institute
Gorard, S., See, B. and Siddiqui, N. (2017). The Trials of Evidence-Based Education. London. Routledge
Greenland, S., Senn, S., Rothman, K., Carlin, J., Poole, C., Goodman, S. and Altman, D. (2016). Statistical Tests, P Values, Confidence Intervals, and Power: A Guide to Misinterpretations. European journal of epidemiology. 31. 4. 337-350.
Jones, G. (2018). Evidence-Based School Leadership and Management: A Practical Guide. London. SAGE Publishing.
Selfridge, R. (2018) Databusting for Schools, London, SAGE
Sharples, J. (2017). A Visions for an Evidence-Based Education System - or Some Things I'd Like to See. London. Education Endowment Foundation
Wasserstein, R. and Lazar, N. (2016). The Asa's Statement on P-Values: Context, Process, and Purpose, the American Statistician, 70:2, 129-133,. The American Statistician. 70. 2. 129-133.

Friday, 20 July 2018

The school research lead, 'factfulness' and 10 reasons why we may be getting things wrong

Over the last few weeks I have been fortunate enough to see both Geoff Barton @RealGeoffBarton and Jill Berry @jillberry speak at conferences.  A theme common to both presentations was the need for school leaders, despite concerns about funding, teacher retention and recruitment to remain optimistic.  And although there are huge pressures of accountability within English schools, school leaders should remember to focus on the joy of teaching young people and the positive change that they can bring to young peoples’ lives.


Now it just so happens that I’m reading the late Hans Rosling’s recently published book: Factfulness: Ten reasons we’re wrong about the world – and why things are better than you think. Rosling argues that when asked simple questions about global trends in the world, people systematically get the answers wrong.  However, the world despite all the challenges we are facing, would appear to be in a much better state than we think.  Unfortunately, we have a tendency to worry about everything all of the time, rather than adopting a world view based on facts.  As a result, we lose our ability to focus on those things that matter most.  Instead, we should adopt a stance of factfulness which involves the stress reducing habit of only carrying opinions for which you have strong supporting facts. 

To help us to adopt a stance of factfulness – Rosling suggests that we adopt 10 rules of thumb

  • Gap – recognizing that when a story talks about a gap – this paints a picture of two separate groups, with a gap in between.  The reality is often not polarised – usually the majority are in the middle – just where the gap is supposed
  • Negativity – recognizing when we get negative news, remembering that information about bad events is much more likely to reach us – when things are getting better we don’t often hear about them
  • Straight line – recognizing the assumptions that a line will just continue straight – and that such lines are rare in reality
  • Fear – remember that frightening things get our attention – and recognising that these things are not necessarily the most risky
  • Size – recognizing when a lonely number seems impressive (small or large) and remembering you could get the opposite impressions if it were compared with or divided by some other relevant number
  • Generalization – recognizing when a category is being used an explanation, and remembering that categories can be divided into sub-categories
  • Destiny – recognizing that many things (including people, religions, cultures, countries) appear to be constant just because change is happening slowly
  • Single – recognizing that a single perspective can limit your imagination
  • Blame – recognizing when a scapegoat is being used and remembering that blaming an individual often steals the focus from other plausible explanations
  • Urgency – recognizing when a decision feels urgent and remembering that it rarely is
So what does this mean for those involved in the leadership of schools?

Rosling et al argue the most important thing that we should be teaching our children is humility and curiosity, and which for me are equally important for school leaders. 

For Rosling et al being humble is all about recognising how difficult it is for your instincts to get out of the way of the ‘facts’. It requires you to be realistic about what you know and don’t know.  It’s about asking yourself the questions – what evidence or new facts would it take to change my mind.  Rosling et al argues this is a far more relaxing stance to take – as you no longer have to be right about everything all of the time.  It’s ok to say, I don’t know, so let’s find out together.

Rosling et al argue being curious means being open to new information, facts, ideas and perspectives.  It demands that you allow new information to challenge your existing ideas, preconceptions and perspectives.  It  requires you to say – you know what, I got that wrong – let’s see what we can learn from this.  But it also requires you to recognise that what you have learnt in the past – be it at university, in teacher-training  or through experience – may well be outdated.  What’s right at the start of your career, and may, 10 or 20 years later be out of date or just plain wrong.

And finally

Please don't think I'm saying all is rosy in English schools and the important trends are all heading in an upward direction.  Rather I'm just asking that you understand how the grounds on claims are being made, the nature of the claim, the warrant for the claims, whether than warrant has strong backing, whether qualifications to the claim, and whether there any rebuttals.

Reference

Rosling, H., Rosling, O. and Rosling Ronnlund, A. (2018). Factfulness: Ten Reasons We're Wrong About the World - and Why Things Are Better Than You Think. London. Sceptre.




Saturday, 14 July 2018

Models of organisational development - What type of culture does your school or college have?


Earlier this week I came across this headline - which although describing further education colleges - I also get the impression that many teachers in schools would recognise similar 'Soviet' style cultures.


So in this post I’m going to write about Patterson, Nolan, et al. (2011) and two models of organisational development - perform or perish and responsive and relational  which were developed from research undertaken in the health sector and which are described in Table 1.

Table 1 Two models of organisational development – Perform or Perish -  Responsive and Relational

Perform or perish
Responsive and relational
Pace: Quick fix, short term, process driven, pushing and fixing

Relational and responsive Complexity: Longer term, focus on
people and perceptions, brokering

External: Top down agenda, local context largely overlooked, off-the- shelf, one-size fits all approaches applied

Locally contextual factors fully acknowledged and addressed, solutions tailored to situation, existing models modified accordingly

Select few determine goals and direction of change
All groups including users/carers involved in deciding goals and direction of change

Punitive and transactional leadership style from top, little unit level leadership

Empowering, inspiring and transformational leadership style at all levels, especially unit

Metrics matter: Superficial, often quantitative targets for success, e.g. patient flow

Meaning matters, relational, dynamic qualitative ‘indicators’ of success, peoples’ experiences

Scored

Profiled
Impoverished change environment results and the ‘senses’ are reduced

Enriched change environment results and the ‘senses’ are enhanced


What’s the relevance for school leadership and management?

Whilst both of these models are ‘idealised’ extreme cases they do provide a useful way for thinking about different approaches to organisational development.  Although for use in the health-care setting; the various dimensions;  be it focus – short-term vs long-term; external demands vs internal context; scale of engagement – the few vs the many; impoverished vs enriched may ring a few alarm bells for colleagues working in schools.  Especially if my Twitter timeline is as anything to go by.  So what are the implications for teachers and school-leaders who wish to bring about evidence-based/informed/enriched schools.

First, the perform or perish model of organisational development seems culturally at odds with developing and evidence-based school.  Punitive and transactional leadership is not going to create the conditions of trust necessary for teachers and schools leaders to challenge existing ways of working.

Second, to develop and evidence-based school will take time, it’s not a quick fix, and will require a recognition of the complexity of the issues to be addressed. It will require multiple iterations of what we think might work to come up with ways of working which are tailored to the particular requirements of a school.

Third, the evidence-based school is not just about teachers accessing, reading and implementing the latest reports of the Education Endowment Foundation.  Just as important, if not more so, is to engage in genuine and meaningful partnerships with pupils, parents and the community.  Their views and perceptions are an integral part of the evidence-based use to help make decisions.

Fourth, no one action or group of actions will lead to a school becoming responsive and relational, it will require a combination of interventions and challenges across the whole-range of the school’s activities, in order to ensure such an approach is part of the organisational DNA of the school.

And finally  

This model and way of thinking about organisational development may be particularly relevant for those in involved in the senior leadership and management of multi-academy trusts and help those leaders avoid the pitfalls of a perform or perish organisational culture.

References

Andrews, N., Gabbay, J., Le May, A., Miller, E., O'Neill, M. and Petch, A. (2015). Developing Evidence Enriched Practice in Health and Social Care with Older People.

Patterson, M., Nolan, M., Rick, J., Brown, J., Adams, R. and Musson, G. (2011). From Metrics to Meaning: Culture Change and Quality of Acute Hospital Care for Older People. Report for National Intsitue for Health Research Service Delivery and Oranisation programme.