New - Distinguishing the practical from the puerile - a simple 2x2 model

If you are school research lead or someone interested in evidence-informed practice and want to be able to identify research with practical relevance, this post is for you.  Using the work of Anderson, Herriot & Hodgkinson (2001) I will describe a simple 2 x 2 model, which uses the dimensions of relevance and rigour, to help you distinguish between - popularist, puerile, pedantic and pragmatic - research.  I will then consider some of the implications for you, either as a school research leads and/or evidence-informed practitioner.

What do we mean by both practical relevance and methodological rigour?

As Anderson et al state, defining precisely what is meant by these terms in not un-problemmatic.  When thinking of relevance, are multitude of questions to ask, for example, for whom is the research relevant for - TAS, NQTS, experienced teachers, HODs, SLTs and headteachers.  Is the research relevant for pupils, parents or other stakeholders within the school.   However, for our purposes, research can be deemed relevant if, in your view, it can directly improve how and what you do in your current role.  As for methodological rigour, well that will all depend upon your view ontological and epistemological standpoint. and I'm certainly not qualified to summarise the debate within a short-blog spot.  That said, regardless of your epistemological or ontological standpoint, the research needs to be a form of 'disciplined inquiry.'  For as Cronbach and Suppes (1969) state:  Disciplined inquiry has a quality that distinguishes it from other sources of opinion and belief. The disciplined inquiry is conducted and reported in such a way that the argument can be painstakingly examined. The report does not depend for its appeal on the eloquence of the writer or on any surface plausibility, (p. 15).

With that in mind, Hood (2003) usefully identifies a number of the characteristics of disciplined inquiry, which include:

• Meaningful topics are addressed
• Systematic, clearly described procedures are employed and described so that readers can follow the logic of the study and assess the validity of the study's conclusion
• There is sensitivity to the errors that are associated with the methods employed and efforts are made to control the errors or consider how they influence the results
• Empirical verification and sound logic are valued: and
• Plausible alternative explanations are considered (p2)

In other words, are the inner workings of the study transparent, with an inherent recognition of the limitations of the study.

Table 1 illustrates  Anderson's et al simple 2 x 2 working model of how to classify research.

Table 1 Combining rigorous practice and relevant research

		Methodological Rigour Low                                        High
Practical Relevance	High	Quadrant 1 ‘Popularist Science’	Quadrant 2 ‘Pragmatic Science’
	Low	Quadrant 4: ‘Puerile Science’	Quadrant 3 ‘Pedantic Science’

Where studies have high levels practical relevance but low levels of methodological rigour, 'Popularist Science' is created and can be found in Quadrant 1.  Examples of this may include research which may be addressing an extremely important and pressing issue, though the studies lack the rigour to warrant any kind of reliance upon their findings.  It could also be used to describe where new educational initiatives or innovations have led to poorly designed studies being conducted, with the need to publish rapidly being more important than the ensuring the quality of the underlying study.  Another attribute of 'popularist science' is the lack of any form of peer-review.    Examples of 'Popularist Science' within education could include those advocating Brain Gym, Learning Styles and possibly Multiple Intelligences.

Where, we have both high practical relevance and methodological rigour, we can describe this as 'Pragmatic Science' and is located in Quadrant 2.   For those interested in closing the gap between researchers and teachers, this is the type of research which should be most common, and be used to influence conversations within staff-rooms.  Examples of this type of research can be found in The Education Endowment Foundation Teaching and Learning Toolkit.  However, there is research which you might think would definitely fall into this category, for example, John Hattie's Visible Learning - where there are significant doubts about the methodological rigour of the research.

As for Quadrant 3, this is where methodological rigour is high and practical relevance is low and can be described as 'Pedantic Science'.  In this situation, we have studies which are almost flawless in design, yet seek to address questions which have little or no relevance to schools, headteachers and teachers.  These studies have a focus on theory not practice, and have been written for a very small group of researchers who specialise in this area.  And I will leave it to you to identify your own favourites in this category.

And finally, Quadrant 4, this is where we have what is termed 'Puerile Science', where researchers have investigated matters of little practical relevance and have done so with research designs, which are sloppy and ill-conceived.

What are the implications of the 2 x 2 model for school research leads and evidence-informed practitioners?

For me there are three main  implications of the model, which need to be considered.

• School research leads are often keen users of social media and read a wide-range of blogs.  Given the very nature of social media, very little if any, of what comes across your timeline will in be in Quadrant 2 i.e relevant and rigorous.  Social media - be it twitter or blogs - should be seen as either entertainment or information rather than education.  In other words, social media may set you off to search out 'Pragmatic Science' but in all likelihood the social media is in itself at best 'Popularist' Quadrant 1 (felt it was a bit of a stretch to call social media science).
• Even if 'Pragmatic Science' can be identified, that does not mean that whatever intervention has been tested, or effect size has been calculated, will necessary work in school or classroom. And remember working out what works is in itself not enough, rather we need to ask the questions :  'how and why does this work and/or not work, for whom, to what extent, in what respects, in what contexts and over what period?'.
• By skilful use of the 2x2 matrix it should be possible to save both yourself and your colleagues a lot to time and resources - by making sure where at all possible you spend your time discussing and implementing interventions which have the characteristics of 'Pragmatic Science.

And finally

T he only real way of knowing whether you have got your classification of 'research' right, is to specialise in a particular field of research.  This blogpost outlines a simple organising framework for thinking not only about academic research, but also other forms of evidence.  As such, it should be as what Daniel Willingham (2012) would describe as a work-around or heuristic - as you seek to bring practically relevant research into your school and classroom.

Reference

Anderson, N., Herriot, P. and Hodgkinson, G.P., 2001. The practitioner‐researcher divide in Industrial, Work and Organizational (IWO) psychology: Where are we now, and where do we go from here?. Journal of Occupational and Organizational Psychology, 74(4), pp.391-411.

Cronbach, L. J., & Suppes, P. (Eds.). (1969). Research for tomorrow’s schools: Disciplined inquiry for education. New York: MacMillan. This is a report of a special committee of the National Academy of Education. It includes a detailed discussion of disciplined inquiry, a number of historical case studies of educational research programs and a set of policy recommendations.

Hood, P. (2003) Scientific Research and Evidence-Based Practice : WestED

Willingham, D. (2012) When Can You Trust The Experts: How to tell good science from bad in education, Jossey-Bass, San Francisco.