Friday, 1 September 2017

Judging the trustworthiness of research findings

In last week's post,  I drew attention to differences in expert opinion over the usefulness of statistical significance testing, particularly as regards randomised controlled trials (RCTs).  This week we will look at what else can go wrong with RCTs and questions that you need to ask when seeking to judge the trustworthiness of the associated research findings.  But first, let's quickly describe what we mean by a RCT

What is a RCT?

(Connolly et al., 2017) describe an RCT as … ‘a trial of particular educational programme or intervention to assess whether it is effective; it is a controlled trial because it compares the progress made by those children taking the programme or intervention with a comparison or control group of children who do not and who continue as normal; and it is randomised because the children have been randomly allocated to the groups being compared (p4)

What can go wrong with RCTs?

Unfortunately, lots can go wrong with RCTs.  (Ginsburg & Smith, 2016)  reviewed 27 RCTs that met the minimum standards of the US based What Works Clearing House,  and found that 26 of the 27 RCTS had serious threats to their usefulness.  These threats are listed below
  • Developer associated. In 12 of the 27 RCT studies (44 percent), the authors had an association with the curriculum’s developer. 
  • Curriculum intervention not well-implemented. In 23 of 27 studies (85 percent), implementation fidelity was threatened because the RCT occurred in the first year of curriculum implementation. The NRC study warns that it may take up to three years to implement a substantially different curricular change.
  • Unknown comparison curricula. In 15 of 27 studies (56 percent), the comparison curricula are either never identified or outcomes are reported for a combined two or more comparison curricula. Without understanding the comparison’s characteristics, we cannot interpret the intervention’s effectiveness. 
  • Instructional time greater for treatment than for control group. In eight of nine studies for which the total time of the intervention was available, the treatment time differed substantially from that for the comparison group. In these studies, we cannot separate the effects of the intervention curriculum from the effects of the differences in the time spent by the treatment and control groups. 
  • Limited grade coverage. In 19 of 20 studies, a curriculum covering two or more grades does not have a longitudinal cohort and cannot measure cumulative effects across grades. 
  • Assessment favors content of the treatment. In 5 of 27 studies (19 percent), the assessment was designed by the curricula developer and likely is aligned in favor of the treatment.
  • Outdated curricula. In 19 of 27 studies (70 percent), the RCTs were carried out on outdated curricula. (Ginsburg & Smith, 2016)(Pii)
So what are you to do?

(Gorard, See, & Siddiqui, 2017) in the recently published book The Trials of Evidence-Based Education suggest the following:

First, check whether there is a clear presentation of research findings - are they presented simply and clearly, with all the relevant data provided.

Second, check whether the research is using effect sizes as the way of presenting the scale of the findings.  If significance testing is being used, and p values are being quoted - you may wish to pause for a moment.  Though remember effect sizes have their own problems (see http://evidencebasededucationalleadership.blogspot.co.uk/2017/01/the-school-research-lead-and-another.html)

Third, check where the research design sits on the research design hierarchy of causal questions. At the top of the hierarchy are studies where participants are randomly allocated between groups; below that are participants matched between groups; below that are naturally occurring groups used; below that is only one group studied and before and after data is used, and finally are case studies used (at the bottom of the hierarchy.

Fourth, check the scale of the study, for example, are at least 100 pupils involved in the study.

Fifth, look out for missing information - how may subjects/participants dropped out of the study.   As a rule of thumb the higher the percentage level of completion of the research, the more trustworthy the findings.  As Gorard at el note - a study with 200 participants and a 100% completion rates is likely to be more trustworthy than a study with 300 participants and a 67% completion rate.

Sixth, check the data quality.   Standardised tests provide higher quality data than say questionnaire data, with impressionistic data for causal questions providing the weakest evidence.  Make sure the outcomes being studied are specified in advance.  Is there likely to be any errors in the data caused by inaccuracy or missing data.

And finally

It's quite easy to be intimidated by quantitative research studies - but if you keep it simple - are effect sizes being used; are subjects randomly allocated between the control and the intervention group; is there missing data; are standardised measures of assessment used; and, are the evaluators clearly separate from the implementors - if the answer to all these questions is yes, there you can have a reasonable expectation that the research findings are trustworthy.

Further reading

Connolly, P., Biggart, A., Miller, S., O'Hare, L., & Thurston, A. (2017). Using Randomised Controlled Trials in Education London: SAGE.

Ginsburg, A., & Smith, M. S. (2016). Do Randomized Controlled Trials Meet the “Gold Standard”? American Enterprise Institute. Retrieved March, 18, 2016.

Gorard, S., See, B., & Siddiqui, N. (2017). The trials of evidence-based education. London: Routledge.





57 comments:

  1. Hurrey! Finally I completed my Graduation and now I will prepare for the Sarkari Naukri 2018 . Many of my friends will be helping me in searching for the Government Jobs 2018

    ReplyDelete
  2. you Blog provide very amazing informative data and it is very amazing for your blog readers Shiksha Mitra News

    ReplyDelete
  3. This article is really informative and useful as far as I concern and keep going. India Post Recruitment

    ReplyDelete
  4. Hello sir, your web site is lovingly serviced and saved as much as date. So it should be, thanks for sharing this with us.
    i also found some helpful sites like yours.


    HSSC jobs
    RPSC jobs
    fresh job alert
    WPSC jobs
    IBPS jobs
    SSC jobs
    West Bengal Police jobs
    RRB recruitment
    RRC recruitment 2018
    Bihar police reruitment
    Bank jobs
    Latest jobs
    UPSSC jobs
    ARMY jobs
    Upcoming jobs

    ReplyDelete
  5. Hi there! If you had to recommend one of the popular erp software options, which one would it be? I heard this one https://ax-dynamics.com/microsoft-dynamics-ax is quite good, but I'd love to find out something more about the alternatives.

    ReplyDelete
  6. This is a wonderful article, Given so much info in it, You can see travel nearby best tourist place Best Honeymoon Destination in this page.Places to visit in Himachal

    ReplyDelete
  7. Hello sir, your web site is lovingly serviced and saved as much as date. So it should be, thanks for sharing this with us.
    i also found some helpful sites like yours. Modicare Products
    Modicare-business-plan
    online-registration-for-modicare-consultant
    Modicare-success-story
    Modicare-achievers-list

    ReplyDelete

  8. Thanks have been given the opportunity to comment. Hopefully what you provided is useful for all those who need them. Visit my website if you want to know more about.
    Latest breaking News in India
    Top Stories News Headlines
    Politics News
    Politics lifestyle
    Actress lifestyle
    New Hindi Movies Celebrity Photos

    ReplyDelete
  9. This is really a great support for the latest adviser from online. Really a supportive work for writing a good article.Any one can enjoy it with the style of writing.
    Organic thai ber farm
    Organic apple farming
    Farming & nursery
    Nursery udhyaan
    Lemon organic farming
    Mosambi organic farming
    Anar organic farming

    ReplyDelete
  10. Thanks have been given the opportunity to comment. Hopefully what you provided is useful for all those who need them. Visit my website if you want to know more about.
    E aadhar card
    Aadhar card download
    Check Aadhar Card Status by name & moblie no.
    e aadhar card update
    Uidai help enquiry
    Aadhar Card Correction

    ReplyDelete

  11. Hey there, You have done an incredible job. I will certainly digg it and in my view suggest to my friends.
    I'm confident they'll be benefited from this website.

    Home Loan Emi Calculator
    Car loan Emi Calculator
    Personal Loan Emi Calculator
    Education Loan Emi Calculators
    Two Wheeler Loan Emi Calculator
    Credit Card Loan Emi Calculator

    ReplyDelete
  12. Much obliged to you for sharing such extraordinary data. It has help me in discovering more insight about Education Loan Emi Calculator

    ReplyDelete
  13. Much obliged to you for sharing such extraordinary data. It has help me in discovering more insight about Emi Calculator For Education Loan

    ReplyDelete
  14. Thank you for sharing this information. It has helped me to know more about
    Educational Loans

    ReplyDelete
  15. The great blog about Emi Calculator Education Loan.Thanks for sharing the blog,seems to be interesting and informative too.Could you help me to find more detail about Emi Calculator Education Loan

    ReplyDelete
  16. Much obliged to you for sharing such extraordinary data. It has help me in discovering more insight about <a href=http://www.avanse.com/education-loan-calculators/emi-calculator
    Emi Calculator For Education Loan

    ReplyDelete
  17. Thank you for sharing this information. It has helped me to know more about Study Loans

    ReplyDelete
  18. Much obliged to you for sharing such extraordinary data. It has help me in discovering more insight about
    Emi Calculator For Education Loan

    ReplyDelete
  19. Read your article its really informative Educational Loan Emi Calculator please visit for details

    ReplyDelete
  20. Nice and interesting information and informative too.
    Can you please let me know the good attraction places we can visit: Education Loan Emi Calculator

    ReplyDelete
  21. Thank you for sharing such great information.
    It has help me in finding out more detail aboutEducation Loan Emi Calculator

    ReplyDelete
  22. Education Loans has really helped students to fulfill their educational goals in life. Also for more such Blog visit - Educational Loan Emi Calculator

    ReplyDelete
  23. Nice and interesting information and informative too.
    Can you please let me know the good attraction places we can visit: Educational Loan Emi Calculator

    ReplyDelete
  24. Thank you for sharing the information, i have got the best information. Educational Loan Calculator are wider concept and helps in building your career so that we can get exciting opportunity in future

    ReplyDelete
  25. I see the greatest contents on your blog and I extremely love reading them.brahmacharyasana

    ReplyDelete
  26. Thank you for sharing the information, i have got the best information. Emi Calculator For Education Loan are wider concept and helps in building your career so that we can get exciting opportunity in future

    ReplyDelete
  27. I think about it is most required for making more on this get engaged

    children education allowance exemption

    ReplyDelete
  28. Thank you for sharing the information, i have got the best information. Education Loan Emi Calculator are wider concept and helps in building your career so that we can get exciting opportunity in future

    ReplyDelete
  29. Thank you for sharing this information. It has helped me to know more about Education Loan For Students

    ReplyDelete
  30. Thank you for sharing this information. It has helped me to know more about Educational Loan Calculator

    ReplyDelete
  31. This is a smart blog. I mean it. You have so much knowledge about this issue, and so much passion. You also know how to make people rally behind it, obviously from the responses.Very informative post ! There is a lot of information here that can help any business get started with a successful social networking campaign !

    best time to visit munnar

    ReplyDelete
  32. I feel really happy to have seen your webpage and look forward to so many more entertaining times reading here. Thanks once more for all the details.This is such a great resource that you are providing and you give it away for free.

    Utkata Konasana

    ReplyDelete
  33. Thank you for sharing this information. It has helped me to know more about Emi Calculator For Education Loan

    ReplyDelete
  34. Its great to know about this blog, Appreciate your efforts with such a valuable information provided. Could you help me to find out more details about Education Loans In India

    ReplyDelete
  35. Thank you for sharing the information, i have got the best information. education loan details

    ReplyDelete
  36. Education loan makes achieving your dreams easier! Also for more such Blog visit - Educational Loan. This will help you remove the barrier and achieve you career goals sooner.

    ReplyDelete
  37. If you're looking for Educational Loan, I would recommend you to check this for more details
    Educational Loan

    ReplyDelete
  38. Awesome work ! I am planning to get an education loan but I was confused about the best deals available for me , your post gave me an amazing idea to explore for higher study in india. Nice post, keep posting.

    ReplyDelete
  39. Awesome work ! I am planning to get an education loan but I was confused about the best deals available for me , your post gave me an amazing idea to explore for student loans repayment. Nice post, keep posting.

    ReplyDelete
  40. This blog is very informative and knowledgable to me. I would love to see such much post from you thank you for sharing. Education Loan Details

    ReplyDelete