Evidence-Based Practice : A Handbook for teachers and school leaders

If you are a school research lead or interested in evidence-based practice, then this post is for you.

Over the last 18 months I have had the opportunity to read, reflect and write about evidence-based practice and education. The more and more I have read about the application of evidence-based practice to schools, the more it has become clear that there are a number of common misconceptions about evidence-informed practice in circulation. In part these misconceptions are the product of a significant body of academic literature and research on evidence-based practice which appears not to be known to many members of the ‘educational academy’ and is resulting in ill-informed discussion and debate about the merits of evidence-based practice. In particular, many commentators on evidence—informed practice have very little detailed awareness of the work being undertaken in the fields of evidence-based medicine and evidence-based management.

With that in mind, I have produced an amended version of the Center for Evidence-Based Management’s booklet “Evidence-Based Management, The Basic Principles”. The booklet written by Denise Rousseau, Rob Briner and Eric Barends, was originally written as a general guide to the principles of evidence-based practice. I have amended the handbook – with the authors’ permission – so the case-studies and examples used, will be more relevant to a school-based audience.  As such, the handbook covers basic questions such as “What is EBP?”, “What counts as evidence?”, “Why do we need EBP?”, “What sources of evidence should be considered?”, “Why do we have to critically appraise evidence?” and “What is the evidence for EBP?”.


If you would like to read more - please go to the following Evidence Based Practice : A handbook for teachers and school leaders

The Biology of School Survival

Last weekend I had the huge privilege of attending the Canons Park Teaching School Alliance conference.  Keven Bartle and Emlyn Lumley gave a fascinating presentation on 'Chaos, complexity and ecological system leaders' and how schools are going to have to respond to an environment of increasing complexity with schools being complex adaptive systems (CAS).   In this post, I will draw upon Reeves, Leving & Ueda's recent article in the Harvard Business Review  (Jan-Feb 2016) to examine in more detail; first, what is meant by a complex adaptive system; second,  how school leaders can attempt to increase resilience within a complex adaptive system; third, some of the limitations of Reeves et al's model when applied to a school-setting.

What is a complex adaptive system?

Reeves at al state that like biological species, companies are 'complex adaptive systems' that continually evolve in hard-to-predict ways.  Local interactions cascade and reshape the entire system; the new structure then influences individual agents - people - resulting in further changes in the system.

They go onto argue that these systems ... are often nested in broader systems.  A populations is a CAS nested in a natural eco-sytems, which itself is nested in a broader biological environment.  So a school is a CAS which is nested within the overall school ecosystem, the education ecosystem, as well as the broader societal system.  As such, not only is there complexity within a school's boundaries, there is tensions between an individual schools and the wider school system-wide ecosystem. As such,what may be good for an individual school may not be good for the broader ecosystem of schools.

Reeves et al argue that there are three broad implications for business leaders, which I will adapt for a school setting.

1. School leaders need to be realistic about what they can predict and control - both within the school and outside of the school. 
2. School leaders need to look beyond their own schools and ensure that their schools contribute positively to the school system as a whole, particularly if they are receiving benefits from the system as a whole.
3. School leaders need to embrace the inconvenient truth that attempts to staff may have negative unintended consequences, which may threaten the school itself.

How can school leaders increase resilience in schools within a complex adaptive system?

Reeves et al identify six principles which can be applied, which may increase the resilience of an organisation (school).  However, they make clear these principles may be in tension with one another and that prioritising one principle may lead to the de-proritisation of another.  As such, Reeves et al argue that the six principles should be seen as a set, which are applied collectively rather than seeing each principle as a standalone strategy or tactic.  With that in mind, we will now examine each of these six principles.

Maintain heterogeneity

School leaders need to ensure their schools are diverse only three axes : people, ideas and endeavours. One way of doing is to hire people from a range of backgrounds, educational qualifications and personality,  and which may be inevitable given the current recruitment crisis.  However, recruiting a diverse range of  individuals, is not enough.  Those individuals need to be given the permission to express their opinions, develop new initiatives and sometimes, to fail.  A safety-first approach - trying to play 'error free football' - may ironically lead to the school becoming even more fragile.

Modularity

In the past there has been much talk about whole-school and integrated approaches. However, it may be necessary to design a school, multi-academy trust or academy chain as a modular system, which will prevent shocks from moving from one part of the system to another.  Yet one of the problems with a modular system is that it potentially gets in the way of networking and collaborative activity.  It may be tempting to increase levels of integration for purposes of integrations, but in increasing interdependence it also leads to an increase in risk.

Redundancy

In systems with redundancy, different aspects of the system often have overlapping roles.  So if one part of the system fails, another part of the system can pick up the slack and carry-out the role.  In the current environment, built in redundancy could be seen as a sign of bloated and inefficient school or organisation.  However, schools are always experiencing shocks - be it new unexpected intitiatives, unanticipated staff turnover or absence -  and there needs to be capacity within the school or MAT to be able to respond to these shocks.

Expect surprise, but reduce uncertainty

In complex adaptive system it is not possible to predict the future shape of the organisation/school.  However, it is possible to spot 'signals', for example, changing patterns of enrolments with feeder schools, changing patterns of pupil applications, changing numbers applications for posts to responsibility - and begin to imagine possible scenarios, and in doing so identify the action to be taken to maximise opportunity and minimise risk.  Reeves et al argue that there are a number of practices which could be adopted:

• Schools may need to accept that a currently successful business model, will at some stage be superseded by some alternative.  Indeed, ways of working that may have been successful in delivering 5 GCSES (inc E&M) at C and above, may not work in the world of the EBacc
• Understand that change comes from the periphery - with incomers having no choice but to do something different to the current dominant players - this can be seen in the way that 'free schools' operate
• Identify the signals from the activities of those organisations and schools who are in effect betting against your continued success - in other words, how well are the scrappy insurgents doing compared to your more established way of working
• Practice contingent thinking - if 'fair funding" becomes a reality - what are you going to do about it. How will it impact upon your way of working and school structures.
• Finally, if possible,  reduce  the threat or idea, by acquiring it or building defences against it.

Now given the context of schools - they key things to do would appear to be to try and spot the signals and engage in contingent thinking

Create feedback loops and adaptive mechanisms

Often the signals that school-leaders may need to be aware that what may be 'hitting' the school, is doing so at some 'organisational' distance from the senior leadership team.  With that in mind - school senior leaders need to engage in real and genuine dialogue with front-line staff - be it school administrators, caretakers, teaching assistants, teachers and others - to keep in in touch with what is happening on the ground.  Second, it's not enough just to get the feedback, it's necessary to act upon it. Reeves talks about out organisations (and schools) having to be ambidextrous - i.e the ability to run the existing school, whilst at the same time reinventing the school in response to the new environment.

Foster trust and reciprocity

Complex adaptive systems require co-operation if they are to survive.  The interests of different schools with a MAT or TSA  or staff who work in them may at times be in conflict.  However, if schools and individuals purse their own interests - then then system as a whole becomes weaker.  Incentives need to be put in place to foster trust and reciprocity.  One way might be ensure that promotion opportunities are given to colleagues who an can not only lead their functional area but contribute to the good of the system as a whole.  The flip side of this - is no matter how 'good' a HOD maybe in leading their department - if this is at the cost of the school as whole, other actions may need to be taken.

Limitations of the model

Keven Bartle kindly reviewed this post before publication and suggested that Reeves et al's model has a number of limitations within a school setting.  Keven argues : One of the inherent tensions in the Reeves approach (and of leveraging complexity in a school environment generally) is that it rather assumes that the leader can see the whole leadership landscape of the school ecology from the outside, as some kind of impartial objective observer and coordinate the very different dynamics to organisational effect. In doing so there is almost the assumption that leadership is a fixed property in charge of the non-fixed properties of agency and interaction and so forth. In doing so it possibly risks reifying the notion of heroic leadership even as it suggests the opposite is true elsewhere in the organisational architecture. 

Perhaps this is inevitable in hierarchical organisations, or maybe there's something that needs to underpin this about the need to de-leader oneself that comes before the "headteacher needs to ensure THEIR schools" bit of the post. It may be fraught with inherent contradictions but such is the nature of chaos and complexity. The key bit before anything else is accepting paradoxes (including the unleading leader). If I follow this logic all the way then it leads to a position that is utterly cooperative in which "OUR school" replaces "MY school" in every element of the hierarchical leadership domains, recruitment (people), professional development (ideas) and strategic planning (actions).  (private email correspondence)

And finally ....

If we accept the notion of the school being a complex adaptive system, being nested in a range of other systems, then this probably has a number of implications for headteachers and senior teams in how they go about their work leading their schools.  Reeves et al may not have all the answers or even the beginning of the questions, but they do provide at the very least is both a framework and agenda for discussion, both within and between schools.

New - Distinguishing the practical from the puerile - a simple 2x2 model

If you are school research lead or someone interested in evidence-informed practice and want to be able to identify research with practical relevance, this post is for you.  Using the work of Anderson, Herriot & Hodgkinson (2001) I will describe a simple 2 x 2 model, which uses the dimensions of relevance and rigour, to help you distinguish between - popularist, puerile, pedantic and pragmatic - research.  I will then consider some of the implications for you, either as a school research leads and/or evidence-informed practitioner.

What do we mean by both practical relevance and methodological rigour?

As Anderson et al state, defining precisely what is meant by these terms in not un-problemmatic.  When thinking of relevance, are multitude of questions to ask, for example, for whom is the research relevant for - TAS, NQTS, experienced teachers, HODs, SLTs and headteachers.  Is the research relevant for pupils, parents or other stakeholders within the school.   However, for our purposes, research can be deemed relevant if, in your view, it can directly improve how and what you do in your current role.  As for methodological rigour, well that will all depend upon your view ontological and epistemological standpoint. and I'm certainly not qualified to summarise the debate within a short-blog spot.  That said, regardless of your epistemological or ontological standpoint, the research needs to be a form of 'disciplined inquiry.'  For as Cronbach and Suppes (1969) state:  Disciplined inquiry has a quality that distinguishes it from other sources of opinion and belief. The disciplined inquiry is conducted and reported in such a way that the argument can be painstakingly examined. The report does not depend for its appeal on the eloquence of the writer or on any surface plausibility, (p. 15).

With that in mind, Hood (2003) usefully identifies a number of the characteristics of disciplined inquiry, which include:

• Meaningful topics are addressed
• Systematic, clearly described procedures are employed and described so that readers can follow the logic of the study and assess the validity of the study's conclusion
• There is sensitivity to the errors that are associated with the methods employed and efforts are made to control the errors or consider how they influence the results
• Empirical verification and sound logic are valued: and
• Plausible alternative explanations are considered (p2)

In other words, are the inner workings of the study transparent, with an inherent recognition of the limitations of the study.

Table 1 illustrates  Anderson's et al simple 2 x 2 working model of how to classify research.

Table 1 Combining rigorous practice and relevant research

		Methodological Rigour Low                                        High
Practical Relevance	High	Quadrant 1 ‘Popularist Science’	Quadrant 2 ‘Pragmatic Science’
	Low	Quadrant 4: ‘Puerile Science’	Quadrant 3 ‘Pedantic Science’

Where studies have high levels practical relevance but low levels of methodological rigour, 'Popularist Science' is created and can be found in Quadrant 1.  Examples of this may include research which may be addressing an extremely important and pressing issue, though the studies lack the rigour to warrant any kind of reliance upon their findings.  It could also be used to describe where new educational initiatives or innovations have led to poorly designed studies being conducted, with the need to publish rapidly being more important than the ensuring the quality of the underlying study.  Another attribute of 'popularist science' is the lack of any form of peer-review.    Examples of 'Popularist Science' within education could include those advocating Brain Gym, Learning Styles and possibly Multiple Intelligences.

Where, we have both high practical relevance and methodological rigour, we can describe this as 'Pragmatic Science' and is located in Quadrant 2.   For those interested in closing the gap between researchers and teachers, this is the type of research which should be most common, and be used to influence conversations within staff-rooms.  Examples of this type of research can be found in The Education Endowment Foundation Teaching and Learning Toolkit.  However, there is research which you might think would definitely fall into this category, for example, John Hattie's Visible Learning - where there are significant doubts about the methodological rigour of the research.

As for Quadrant 3, this is where methodological rigour is high and practical relevance is low and can be described as 'Pedantic Science'.  In this situation, we have studies which are almost flawless in design, yet seek to address questions which have little or no relevance to schools, headteachers and teachers.  These studies have a focus on theory not practice, and have been written for a very small group of researchers who specialise in this area.  And I will leave it to you to identify your own favourites in this category.

And finally, Quadrant 4, this is where we have what is termed 'Puerile Science', where researchers have investigated matters of little practical relevance and have done so with research designs, which are sloppy and ill-conceived.

What are the implications of the 2 x 2 model for school research leads and evidence-informed practitioners?

For me there are three main  implications of the model, which need to be considered.

• School research leads are often keen users of social media and read a wide-range of blogs.  Given the very nature of social media, very little if any, of what comes across your timeline will in be in Quadrant 2 i.e relevant and rigorous.  Social media - be it twitter or blogs - should be seen as either entertainment or information rather than education.  In other words, social media may set you off to search out 'Pragmatic Science' but in all likelihood the social media is in itself at best 'Popularist' Quadrant 1 (felt it was a bit of a stretch to call social media science).
• Even if 'Pragmatic Science' can be identified, that does not mean that whatever intervention has been tested, or effect size has been calculated, will necessary work in school or classroom. And remember working out what works is in itself not enough, rather we need to ask the questions :  'how and why does this work and/or not work, for whom, to what extent, in what respects, in what contexts and over what period?'.
• By skilful use of the 2x2 matrix it should be possible to save both yourself and your colleagues a lot to time and resources - by making sure where at all possible you spend your time discussing and implementing interventions which have the characteristics of 'Pragmatic Science.

And finally

T he only real way of knowing whether you have got your classification of 'research' right, is to specialise in a particular field of research.  This blogpost outlines a simple organising framework for thinking not only about academic research, but also other forms of evidence.  As such, it should be as what Daniel Willingham (2012) would describe as a work-around or heuristic - as you seek to bring practically relevant research into your school and classroom.

Reference

Anderson, N., Herriot, P. and Hodgkinson, G.P., 2001. The practitioner‐researcher divide in Industrial, Work and Organizational (IWO) psychology: Where are we now, and where do we go from here?. Journal of Occupational and Organizational Psychology, 74(4), pp.391-411.

Cronbach, L. J., & Suppes, P. (Eds.). (1969). Research for tomorrow’s schools: Disciplined inquiry for education. New York: MacMillan. This is a report of a special committee of the National Academy of Education. It includes a detailed discussion of disciplined inquiry, a number of historical case studies of educational research programs and a set of policy recommendations.

Hood, P. (2003) Scientific Research and Evidence-Based Practice : WestED

Willingham, D. (2012) When Can You Trust The Experts: How to tell good science from bad in education, Jossey-Bass, San Francisco.