Friday 17 August 2018

The school research lead and RCTs - what can a systematic review tell us?

As a school research lead one of the things that you will have to get grips with is the debate over whether randomised controlled trials (RCTs) can make any meaningful contribution to understanding ‘what works’ in educational settings.  Helpfully  Connolly, Keenan, et al. (2018)  have recently had published systematic review on the use of RCTs in education, which seeks to address four key criticisms of RCTs: it is not possible to undertake RCTs in education; RCTs are blunt research designs that ignore context and experience; RCTs tend to generate simplistic universal laws of ‘cause and effect’; and that they are inherently descriptive and contribute little to theory. So in this rest of this post I will provide extracts from the systematic review, examine the review’s answer to the questions posed, identify some missed opportunities, and finally, make some comments about RCTS and the work of school research leads.

Connolly, et al. (2018) systematic review - extracts

The systematic review found a total of 1017 unique RCTs that have been completed and reported between 1980 and 2016, with three quarters of these being produced over the last 10 years.

Over half of all RCTs identified were conducted in North America and a little under a third in Europe. 

The RCTs cover a wide range of educational settings and focus on an equally wide range of educational interventions and outcomes. 

Connolly et al go onto argue that the review: provides clear evidence to counter the claim that it is just not possible to do RCTs in education. As has been demonstrated, there now exist over 1000 RCTs that have been successfully completed and reported across a wide range of educational settings and focusing on an equally wide range of interventions and outcomes. Whilst there is a clear dominance of RCTs from the United States and Canada, there are significant numbers conducted across Europe and many other parts of the world. Many of these have been relatively large-scale trials, with nearly a quarter (248 RCTs in total) involving over one thousand participants. Moreover, a significant majority of the RCTs identified (80.8%) were able to generate evidence of the effects of the educational interventions under investigation.

As noted earlier, these  figures are likely to be under-estimates given the limitation of the present systematic review, with its restricted focus on articles and reports published in English. Nevertheless, the evidence is compelling that it is quite possible to undertake RCTs in educational settings. Indeed, across the 1017 RCTs identified through this systematic review, there are almost 1.3 million people that have participated in an RCT within an education setting between 1980 and 2016 ……

Secondly, there is some evidence to counter the criticism that RCTs ignore context and experience. Whilst they only constitute a minority of the trials identified (37.7%), there were 381 RCTs found that included a process evaluation component…..

Thirdly, there is more evidence to suggest that the RCTs produced within the time period
have attempted to avoid the generation of universal laws of ‘ cause and effect’ . Certainly, those RCTs identified that have included at least some subgroup analyses would suggest a more nuanced approach amongst those conducting RCTs, that acknowledges that educational interventions are not likely to have the same effect across all contexts and all groups of students. Moreover, this is clearly evident amongst the majority of RCTs reported (77.9%) that included at least some discussion of and reflections on the limitations of the findings in terms of their generalizability….

…. (in) relation to the fourth criticism regarding the a theoretical nature of RCTs, this is also challenged to some extent by the findings presented above. A clear majority of RCTs that were reported included some discussion of the theory underpinning the interventions under investigation (77.3%). Moreover, a majority of RCTs (60.5%) also provided some reflections on the implications of their findings for theory….

Overall, the findings from this systematic review of RCTs undertaken in education 1980 – 2016 are mixed. On the one hand, there is clear evidence that it is possible to conduct RCTs in education, regardless of the nature of the education setting or of the particular type and focus of the intervention under consideration….

On the other hand, it is perhaps not surprising that criticisms of RCTs continue when nearly two thirds of RCTs in this period of time have not included a process evaluation component and where nearly half of them have not looked beyond the overall effects of the intervention in question for the sample as a whole. Similarly, it is difficult to challenge the view that RCTs promote a simplistic and atheoretical approach to educational research when nearly 40% of trials in this analysis have failed to reflect upon the implications of their findings for theory.

Have the criticisms of RCTs been answered?

The main assumption of the systematic review is that it is possible to confirm whether RCTS can be carried out in education by merely counting the number of RCTs – subject to certain criteria – have been carried out.  However, this does not give you an answer as to the question as to whether it’s possible to carry out RCTS in an educational context.  It merely tells you, that a number of RCTS have been carried out by researchers and commissioners of research who believe it is possible to carry out RCTS within education.  

Missed opportunities

One of the first things that struck me about the systematic review was that it appeared to give very little attention, if any, as to whether any of the RCTs had systematic flaws in their design.  It may well be that over a 1000 RCTs have been conducted between 1980 and 2016 but a recent review of RCTs by Ginsburg and Smith (2016) reported that 27 RCTs that met the minimum standards of the What Works Clearinghouse based in the US, found that 26 of the RCTS had serious threats to their usefulness.  This would suggest that may be only 40 or so of the RCTS included in Connolly’s review did not have some kind of the serious threat to their trustworthiness.  Second, we then need to look at how many of these 40 or so RCTs included a process evaluation, made a contribution to theory and did not seek to overgeneralise.  It’s likely to a be very small percentage of the number RCTS which had been carried out. In other words, out of 1000 plus RCTs how many did not have fundamental flaws in design or some other failing.

What does this all mean for you as a school research lead?

For a start, Connolly et al provide an accessible introduction to a discussion of the issues associated with RCTs.  As such, it’s worth spending time reading the review.

Second, the review highlights some of the issues associated with the hierarchies of evidence.  Both systematic reviews and RCTS appear to be near the top of hierarchies of evidence.  However, what matters more is whether the research design is applicable to the research question at hand, Gorard, See, et al. (2017) and Sharples (2017).

Third, when reading RCTs it is necessary to have some kind of check-list so that you judge the trustworthiness of what you are reading.  In my forthcoming book, Evidence-Based School Leadership: A practical guide,  Jones (2018) - I try and do this by combing the work of Gorard, et al. (2017) and Ginsburg and Smith (2016).  For example, look out for whether the intervention is being evaluated by the researchers who developed the intervention.

Fourth, setting aside the issue of whether it’s possible to conduct an RCT within education, assuming that it is, theirs is the issue about how the analysis of the results is carried out, and whether p-values and statistical significance have been misused.  This is not the time or place to go into detail about this debate – that said I would recommend you have a look at Wasserstein and Lazar (2016) – and the American Statistical Association’s statement on p-values - Gorard, et al. (2017) with more adventurous readers having a look at Greenland, Senn, et al. (2016).  If on the other hand, you fancy having a look at something which is far more accessible, I recommend you have a look at Richard Selfridge’s new book – Databusting for Schools.

And finally

RCTS within educational research are not going away any time in the near future and I hope this post has provided a little more clarity

References

Connolly, P., Keenan, C. and Urbanska, K. (2018). The Trials of Evidence-Based Practice in Education: A Systematic Review of Randomised Controlled Trials in Education Research 1980–2016. Educational Research.
Ginsburg, A. and Smith, M. (2016). Do Randomized Controlled Trials Meet the “Gold Standard”? American Enterprise Institute
Gorard, S., See, B. and Siddiqui, N. (2017). The Trials of Evidence-Based Education. London. Routledge
Greenland, S., Senn, S., Rothman, K., Carlin, J., Poole, C., Goodman, S. and Altman, D. (2016). Statistical Tests, P Values, Confidence Intervals, and Power: A Guide to Misinterpretations. European journal of epidemiology. 31. 4. 337-350.
Jones, G. (2018). Evidence-Based School Leadership and Management: A Practical Guide. London. SAGE Publishing.
Selfridge, R. (2018) Databusting for Schools, London, SAGE
Sharples, J. (2017). A Visions for an Evidence-Based Education System - or Some Things I'd Like to See. London. Education Endowment Foundation
Wasserstein, R. and Lazar, N. (2016). The Asa's Statement on P-Values: Context, Process, and Purpose, the American Statistician, 70:2, 129-133,. The American Statistician. 70. 2. 129-133.

6 comments:

  1. This is how my partner Wesley Virgin's adventure launches in this SHOCKING and controversial VIDEO.

    You see, Wesley was in the army-and soon after leaving-he unveiled hidden, "self mind control" tactics that the government and others used to get anything they want.

    THESE are the exact same methods tons of celebrities (notably those who "became famous out of nothing") and top business people used to become wealthy and successful.

    You've heard that you use only 10% of your brain.

    That's really because most of your brain's power is UNTAPPED.

    Perhaps this expression has even occurred IN YOUR own mind... as it did in my good friend Wesley Virgin's mind about seven years back, while driving a non-registered, beat-up garbage bucket of a car without a driver's license and $3.20 in his bank account.

    "I'm absolutely fed up with living check to check! When will I get my big break?"

    You've taken part in those types of questions, ain't it so?

    Your success story is waiting to start. All you have to do is in YOURSELF.

    Learn How To Become A MILLIONAIRE Fast

    ReplyDelete
  2. This blog is the general information for the feature. You got good work for this blog. We have a developing our creative content of this mind. Thank you for this blog. This for very interesting and useful.

    IB & British Curriculum School in Al Khail Gate, Dubai


    ReplyDelete
  3. Pro Dissertation Writing provide best doctoral dissertation writing services online in the UK. Our expert PhD dissertation writing service will provide assistance related to doctoral dissertation.

    ReplyDelete
  4. Extremely impressed with the way you have played with your words in the above write-up. I appreciate your hard work. If you are among the students who are looking for a MYOB Assignment Help platform that can provide top quality solutions to them. It is a reliable academic writing help platform that ensures you submit nothing but the best to your academic institutions and fetch the highest possible grades.

    ReplyDelete
  5. Australia is the largest country in Oceania and one of the major countries in the world. So naturally, Australia is the birthplace of some of the most prestigious colleges and universities globally. Australia provides some of the best university assignment help through the excellent faculties and universities.
    Students prefer Australia to pursue their master's degrees or Post Doctorate degrees. Apart from the excellent teaching staff in universities, they have many options to do my homework as well. In addition, the country provides a beautiful landscape and is a gastronomic delight for the foodies. So, that is a lucrative proposition for many students who are serious about study help and making a career in this country.
    Curtin University
    Founded more than five decades ago, in 1966, Curtin University is located in the beautiful suburban setting of Perth. It is a non-profit public higher education university with multiple branch campuses. They have been officially recognised by the Department of Education and The University of Western Australia
    This is one of the oldest universities in Australia. Since its inception in 1911, The University of Western Australia is also a non-profit higher education institute located in Crawley. They have some of the best faculties among all the universities in Western Australia and are famous for providing immaculate do my essay to all their students.
    Murdoch University
    Murdoch is a tiny city in the heart of Western Australia. Murdoch University was founded in this very city back in the year 1975. They are also recognised by the Department of Education and Training of Australia and were always a coeducational university. They offer a bunch of courses and programs in several subject areas for instant assignment help. Students never have a problem finding the subject of their choice from the variety of options Murdoch University provides.
    Edith Cowan University
    This university is a relatively newer university and was founded in 1991. Edith Cowan University is situated in the small city of Joondalup in Western Australia. This is also a public university focusing on higher education and works on a non-profit model.
    The University of Notre Dame Australia
    UNDA or The University of Notre Dame Australia is the only private education institute on this list. Established in 1989, this university is situated in the chic urban backdrop of Fremantle in Western Australia.
    We have a few more services: 101586 Assignment Answers| 104801B Assessment Answers| 11408 Task Answers| 11485 Solutions| 11641 Answers

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete