Friday 1 September 2017

Judging the trustworthiness of research findings

In last week's post,  I drew attention to differences in expert opinion over the usefulness of statistical significance testing, particularly as regards randomised controlled trials (RCTs).  This week we will look at what else can go wrong with RCTs and questions that you need to ask when seeking to judge the trustworthiness of the associated research findings.  But first, let's quickly describe what we mean by a RCT

What is a RCT?

(Connolly et al., 2017) describe an RCT as … ‘a trial of particular educational programme or intervention to assess whether it is effective; it is a controlled trial because it compares the progress made by those children taking the programme or intervention with a comparison or control group of children who do not and who continue as normal; and it is randomised because the children have been randomly allocated to the groups being compared (p4)

What can go wrong with RCTs?

Unfortunately, lots can go wrong with RCTs.  (Ginsburg & Smith, 2016)  reviewed 27 RCTs that met the minimum standards of the US based What Works Clearing House,  and found that 26 of the 27 RCTS had serious threats to their usefulness.  These threats are listed below
  • Developer associated. In 12 of the 27 RCT studies (44 percent), the authors had an association with the curriculum’s developer. 
  • Curriculum intervention not well-implemented. In 23 of 27 studies (85 percent), implementation fidelity was threatened because the RCT occurred in the first year of curriculum implementation. The NRC study warns that it may take up to three years to implement a substantially different curricular change.
  • Unknown comparison curricula. In 15 of 27 studies (56 percent), the comparison curricula are either never identified or outcomes are reported for a combined two or more comparison curricula. Without understanding the comparison’s characteristics, we cannot interpret the intervention’s effectiveness. 
  • Instructional time greater for treatment than for control group. In eight of nine studies for which the total time of the intervention was available, the treatment time differed substantially from that for the comparison group. In these studies, we cannot separate the effects of the intervention curriculum from the effects of the differences in the time spent by the treatment and control groups. 
  • Limited grade coverage. In 19 of 20 studies, a curriculum covering two or more grades does not have a longitudinal cohort and cannot measure cumulative effects across grades. 
  • Assessment favors content of the treatment. In 5 of 27 studies (19 percent), the assessment was designed by the curricula developer and likely is aligned in favor of the treatment.
  • Outdated curricula. In 19 of 27 studies (70 percent), the RCTs were carried out on outdated curricula. (Ginsburg & Smith, 2016)(Pii)
So what are you to do?

(Gorard, See, & Siddiqui, 2017) in the recently published book The Trials of Evidence-Based Education suggest the following:

First, check whether there is a clear presentation of research findings - are they presented simply and clearly, with all the relevant data provided.

Second, check whether the research is using effect sizes as the way of presenting the scale of the findings.  If significance testing is being used, and p values are being quoted - you may wish to pause for a moment.  Though remember effect sizes have their own problems (see http://evidencebasededucationalleadership.blogspot.co.uk/2017/01/the-school-research-lead-and-another.html)

Third, check where the research design sits on the research design hierarchy of causal questions. At the top of the hierarchy are studies where participants are randomly allocated between groups; below that are participants matched between groups; below that are naturally occurring groups used; below that is only one group studied and before and after data is used, and finally are case studies used (at the bottom of the hierarchy.

Fourth, check the scale of the study, for example, are at least 100 pupils involved in the study.

Fifth, look out for missing information - how may subjects/participants dropped out of the study.   As a rule of thumb the higher the percentage level of completion of the research, the more trustworthy the findings.  As Gorard at el note - a study with 200 participants and a 100% completion rates is likely to be more trustworthy than a study with 300 participants and a 67% completion rate.

Sixth, check the data quality.   Standardised tests provide higher quality data than say questionnaire data, with impressionistic data for causal questions providing the weakest evidence.  Make sure the outcomes being studied are specified in advance.  Is there likely to be any errors in the data caused by inaccuracy or missing data.

And finally

It's quite easy to be intimidated by quantitative research studies - but if you keep it simple - are effect sizes being used; are subjects randomly allocated between the control and the intervention group; is there missing data; are standardised measures of assessment used; and, are the evaluators clearly separate from the implementors - if the answer to all these questions is yes, there you can have a reasonable expectation that the research findings are trustworthy.

Further reading

Connolly, P., Biggart, A., Miller, S., O'Hare, L., & Thurston, A. (2017). Using Randomised Controlled Trials in Education London: SAGE.

Ginsburg, A., & Smith, M. S. (2016). Do Randomized Controlled Trials Meet the “Gold Standard”? American Enterprise Institute. Retrieved March, 18, 2016.

Gorard, S., See, B., & Siddiqui, N. (2017). The trials of evidence-based education. London: Routledge.





41 comments:

  1. you Blog provide very amazing informative data and it is very amazing for your blog readers Shiksha Mitra News

    ReplyDelete
  2. This article is really informative and useful as far as I concern and keep going. India Post Recruitment

    ReplyDelete
  3. Hi there! If you had to recommend one of the popular erp software options, which one would it be? I heard this one https://ax-dynamics.com/microsoft-dynamics-ax is quite good, but I'd love to find out something more about the alternatives.

    ReplyDelete
  4. This is a wonderful article, Given so much info in it, You can see travel nearby best tourist place Best Honeymoon Destination in this page.Places to visit in Himachal

    ReplyDelete
  5. The great blog about Emi Calculator Education Loan.Thanks for sharing the blog,seems to be interesting and informative too.Could you help me to find more detail about Emi Calculator Education Loan

    ReplyDelete
  6. Much obliged to you for sharing such extraordinary data. It has help me in discovering more insight about <a href=http://www.avanse.com/education-loan-calculators/emi-calculator
    Emi Calculator For Education Loan

    ReplyDelete
  7. Thank you for sharing this information. It has helped me to know more about Study Loans

    ReplyDelete
  8. Read your article its really informative Educational Loan Emi Calculator please visit for details

    ReplyDelete
  9. Thank you for sharing such great information.
    It has help me in finding out more detail aboutEducation Loan Emi Calculator

    ReplyDelete
  10. Education Loans has really helped students to fulfill their educational goals in life. Also for more such Blog visit - Educational Loan Emi Calculator

    ReplyDelete
  11. Nice and interesting information and informative too.
    Can you please let me know the good attraction places we can visit: Educational Loan Emi Calculator

    ReplyDelete
  12. Thank you for sharing the information, i have got the best information. Educational Loan Calculator are wider concept and helps in building your career so that we can get exciting opportunity in future

    ReplyDelete
  13. I see the greatest contents on your blog and I extremely love reading them.brahmacharyasana

    ReplyDelete
  14. Thank you for sharing the information, i have got the best information. Emi Calculator For Education Loan are wider concept and helps in building your career so that we can get exciting opportunity in future

    ReplyDelete
  15. I think about it is most required for making more on this get engaged

    children education allowance exemption

    ReplyDelete
  16. Thank you for sharing the information, i have got the best information. Education Loan Emi Calculator are wider concept and helps in building your career so that we can get exciting opportunity in future

    ReplyDelete
  17. Thank you for sharing this information. It has helped me to know more about Education Loan For Students

    ReplyDelete
  18. Thank you for sharing this information. It has helped me to know more about Educational Loan Calculator

    ReplyDelete
  19. This is a smart blog. I mean it. You have so much knowledge about this issue, and so much passion. You also know how to make people rally behind it, obviously from the responses.Very informative post ! There is a lot of information here that can help any business get started with a successful social networking campaign !

    best time to visit munnar

    ReplyDelete
  20. I feel really happy to have seen your webpage and look forward to so many more entertaining times reading here. Thanks once more for all the details.This is such a great resource that you are providing and you give it away for free.

    Utkata Konasana

    ReplyDelete
  21. Thank you for sharing this information. It has helped me to know more about Emi Calculator For Education Loan

    ReplyDelete
  22. Its great to know about this blog, Appreciate your efforts with such a valuable information provided. Could you help me to find out more details about Education Loans In India

    ReplyDelete
  23. If you're looking for Educational Loan, I would recommend you to check this for more details
    Educational Loan

    ReplyDelete
  24. This blog is very informative and knowledgable to me. I would love to see such much post from you thank you for sharing. Education Loan Details

    ReplyDelete
  25. All information about the Sarkari Rojgar Result can be found on www.examsresult.in

    ReplyDelete
  26. Unfortunately, lots can go wrong with RCTs. (Ginsburg & Smith, 2016) reviewed 27 RCTs that met the minimum standards of the US based What Works Clearing House, and found that 26 of the 27 RCTS had serious threats to their usefulness. These threats are listed below
    best website developers vancouver ,
    web development services in canada ,

    ReplyDelete
  27. how to check result Are you getting handy information to prepare for government exams. Are you facing any kind of problem in preparing for your competitive exams? If you have doubts related to exam results and government exams? For all these information, you visit My Sarkari Result Service Provider

    Also Visit
    www.mysarkariresult.co.in

    ReplyDelete
  28. This information is helpful! Your post gave us an amazing idea about the best deals for Education Loan Eligibility Calculator in the market.

    ReplyDelete
  29. It was an amazing blog to better understand the importance of loans for abroad education. Learn more about to understand better about Educational Loan Calculator.

    ReplyDelete
  30. Are you getting handy information to prepare for sarkari exam 2022 date. Are you facing any kind of problem in preparing for your competitive exams? If you have doubts related to exam results and government exams? For all these information, you visit My Sarkari Result Service Provider

    ReplyDelete
  31. Thank you for sharing this blog. This is very informative and useful for me. Please share more blogs. Thank you.
    student loans New Zealand

    ReplyDelete
  32. Blog was amazing and worth a while.
    student loan to study in usa

    ReplyDelete
  33. Are you looking for the Latest West Bengal Jobs for this week, West Bengal School Service Commission has released a latest employment notification. There are vacancies under Group C & Group D Posts. Candidates who were looking from a long time to apply now have opportunity to for West Bengal SSC Jobs. Important details like educational qualification, age limit, selection process, admit card, application fee, online application form etc are mentioned at online abedon blog. The online application form is available at the official website.

    ReplyDelete