The School Research Lead and Setting up a Journal Club

In my last post I began to explore the use by School Research Leads of Journal Clubs to help colleagues engage in the critical appraisal of texts.  In this week's post I will again be using the work of Straus et al (2011) in the realm of evidence-based medicine to help School Research Leads set up and successfully run a Journal Club.

In setting up and running a Journal Club there would appear to be three broad sets of issues to address.

1. What are the learning goals of the Journal Club?
2. Strategies to support the introduction of the Journal Club?
3. How should an individual Journal Club session be conducted?

What are the learning goals of the Journal Club

There are at least three core learning goals for a Journal Club

• Learning about the best evidence available to support the needs of pupils/staff in addressing a particular learning need.
• Learning about important new evidence that may lead to a change in pedagogy, systems or approach to leadership and management
• Developing skills at being better consumers of research in order to become more effective evidence-based practitioners.

None of the above are mutually exclusive, although the focus will very much depend upon the needs of the pupils, department, school or individual member of staff.

Strategies to support the setting up of the Journal Club 

There are a number of different strategies and tactics that can be used to make the Journal Club effective and these include:

• Building a coalition of senior colleagues and peers who will help you achieve your aims.
• Securing some appropriate resources to allow the Journal Club - be it a small-budget to access materials, tea, biscuits, a decent room and time slot which is not 4.30pm on a Friday afternoon.
• Be clear that this is a collaborative endeavour which will engage colleagues in group learning activities.
• Preparing some basic materials for the first few sessions e.g.  materials which would easily lend themselves to a critical appraisal.
• Developing your own skills in facilitating discussions with colleagues.

Running a Journal Club session 

Whether all of the following can be accomplished in a single session will depend upon the time and other resources which have been made available. Indeed, it may be possible to run a cycle of meetings which where in any one meeting only two of the three components are covered.

• Part One - establish the learning needs to be covered in the future - be it a specific pedagogic, assessment or leadership and management issue (please note I don't regard research/evidence-based practice the exclusive preserve of front-line teachers).  This could involve accessing existing evidence or reviewing new evidence which has been published.  Alternatively, a colleague may have specific research/evidence-based practice skill issue which they wish to develop.
• Part Two - this involve the sharing of the outcomes of the search for evidence which was based on the learning need identified in the previous session - this can take the form of photocopies of summaries, original articles or other evidence.  Remember, evidence-based practice does not just rely on 'scientific' research evidence, but also draws upon internal school data and the views of stakeholders.
• Part Three - this is the main part of the session and involves the critical appraisal of the evidence presented.  As mentioned last week this might involve discussions around:
• Why are we reading this?
• What are the authors trying to do in writing this?
• What are the authors saying that is relevant to what we want to find out?
• How convincing is what the authors are saying?
• What can we make of this? (adapted from Wallace and Wray, 2011)

Of course, this is not a strict menu or template about how to run and conduct a Journal Club. On the other hand, I do hope it provides some food for thought and a useful stimulus in how to run effective sessions.   Finally, if I was to offer one piece of advice it is essential to make sure colleagues have sufficient access to articles, evidence or books etc - for without that -  all you do is meet with frustration.

References

Straus, S.E., Glasziou, P., Richardson, W. S. & Haynes, B.R. (2011)  Evidence Based Medicine : How to practice and teach it, (4th edition), Churchill Livingston.
Wallace, M. and Wray, A. (2011) Critical Reading and Writing for Postgraduates (2nd edition), Sage, London

The School Research Lead and Journal Clubs - Writing a Critical Synopsis of a Text

In my recent posts on the role of the school research lead I have argued that one of the key tasks of such a role is to help colleagues become critical consumers of research.  Indeed, the critical appraisal of evidence, both research texts and other sources -is an essential component of becoming an effective evidence-based practitioner.  One of the ways in which these skills can be developed is through is the creation of a Journal Club, where colleagues regularly get together to critically review journal articles.  In the rest of this post I will look at one particular method of developing skills in the critical appraisal of evidence, which can be used as an integral part of a Journal Club or independently by colleagues looking to become more critical consumers of research.

WRITING A CRITICAL SYNOPSIS
Mike Wallace and Alison Wray in their book Critical Reading and Writing for Postgraduates (second edition) identify 5 basic questions which form the basis of a critical synopsis, which will also form the starting point for a subsequent critical analysis. The FIVE basic questions are:

1. Why am I reading this?
Am I trying to formulate an answerable question.? Is this reading going to help me gain a better understanding of background or foreground questions?  Is the reading for school-based action or for the purpose of academic study? Is this reading about helping me develop my skills as a critical reader.

2. What are the authors trying to do in writing this?

The authors my have a range of differing purposes including : criticising policy and practice; providing a synthesis of others' work in the area; reporting on their own research activities; or they maybe engaged in theory development.

3. What are the authors saying that is relevant to what I want to find out?

What is the text actually about and what do the authors say about it?  How does text the relate to your own interests: is the text directly linked to your interests or key question(s); is the text indirectly related to your key interests/questions; or does the text provide a tangential but interesting perspective on the issues/questions at hand?

4. How convincing is what the authors are saying?

How has the data been collected and analysed?  Does the data directly lead to the analysis or conclusion being drawn, or are there 'leaps' in logic of the argument?

5. What can I make of this?

Is this text worthy of further of further reference or deeper analysis, or is it something which will only be referred to in passing.  Alternatively, is the text interesting but on the other hand not directly relevant to your core interests.

At first, this process may appear to make the reading of appropriate texts more time consuming.  On the other hand, by going through a structured process it make the reading itself more beneficial. In the beginning, it may be necessary to produce a written critical synopsis,  particularly if the reading is being undertaken for more formal processes, such as preparation for the writing of an academic essay or dissertation.

To conclude, being a school-research lead or evidence-based practitioner is about developing the skills to ask better questions, and in doing so become a more proficient consumer of educational research.  A Journal Club which engages in the critical appraisal of a text or text(s) is one approach which can assist colleagues in meeting in this aim.  In my next two posts I will look in more detail about the mechanics of running a successful Journal Club.

USEFUL RESOURCES
A link to resources produced by Wallace and Wray which will help you produce a critical synopsis can be found using this link.

The School Research Lead and Strategies that Work - Lessons from Evidence-Based Medicine

In this post I will be drawing upon work undertaken in the field of Evidence-Based Medicine  to identify a number of strategies will increase the likelihood that the work of the school research lead will be successful.  In doing so, I will be adapting the work of Straus et al (2011) for use in a school-setting.

The work of the school-research lead is most likely to be successful when the work:

Centres around real teaching and learning decisions
It's important to ensure that any research or evidence-based practice focus on the needs of the pupils/department/ and school.  Ideally, by focussing on our pupils' needs we can begin to formulate the right questions and then engage in an appropriate search strategy to identify relevant sources of evidence.   Once, the various sources of evidence have found and appraised it's important to come back to the needs of the pupil to inform the stance on how to proceed.

Focuses on the needs to teachers (and leaders) to become better practitioners
Many colleagues will be relatively new to both and research and evidence-based practice and it is important to start with where those colleagues are in their own professional journey rather than start with where you would like or expect them to be.  Engaging in this work will involve some 'risk' and it is important to recognise colleagues will have both different levels of teacher and research expertise and appetite for risk.

Connects  'old' to 'new; knowledge
Most colleagues who engage in this work will already have a very high level of teacher expertise and this provides opportunities for colleagues to reflect on that experience and how that may enable them to address gaps of knowledge and understanding when meeting future challenges in meeting the needs of pupils.

Involves everyone in the teaching-team
Many teaching and learning challenges require the input of all members of the team and department, be it teachers, teaching assistants, technicians and volunteers.  All will have experience or evidence which can they bring to the 'table' and lead to more informed decisions.  Indeed, they may identify the need to change their practice, with that in mind it is best to actively engage with them right from the very beginning of the process.

Matches the school, time and other circumstances
It's important to ensure that any research or evidence-based practice development activity is developed with the particular circumstances of a school-mind.  The amount of CPD time which is made available will largely determine what can be done.  On the one hand, if 'research' and evidence-based practice becomes the school's number one CPD priority that will obviously impact on the scale and ambition of the activities undertaken.  On the other hand, if this is not the case, then less resource hungry activities may be required, for example, work-shops on how to formulate answerable questions using the PICO or CIMO formats or how to critically appraise a research text.

Matches opportunity
There is an old saying that luck equals preparation meeting opportunity.  It maybe that the time is not right for a research or evidence-based activity within your school.  That does not mean time cannot be spent developing both knowledge, understanding and skills so that when the appropriate opportunity arises, be it in a 1 to 1 conversation, departmental staff meeting or other setting the use of evidence-based practice techniques can be used.

Is explicit about how to make judgements and the integration of other sources of evidence from -  personal, school or stakeholders - into the decision-making process
It is absolutely essential to emphasise this is not about 'research evidence' replacing the expertise of teachers.  Rather it is about ensuring the use of the best available evidence to inform judgments - be it about teaching or managerial issues.

Increases the capacity of teachers (and leaders) to learn more in the future
Any work in this area should be designed to increase the capacity of individuals to undertake this type of work for themselves in the future.  It is important to recognise that not staff want to have a PhD, or even if they did that they have the range of support require to achieve it.  Much of this work is about helping colleagues become better consumers of research so that they can become better practitioners - be it as teachers and or leaders.

Finally, I hope colleagues has found this post useful and in my next two posts I discuss two relatively straightforward techniques that can be used by school research-leads to help colleagues become better evidence-based practitioners.

Reference

Straus, S.E., Glasziou, P., Richardson, W. S. & Haynes, B.R. (2011)  Evidence Based Medicine : How to practice and teach it, (4th edition), Churchill Livingston

The school research lead and pitfalls to be avoided

One of the challenges of the newly emerging role of the school research lead is that to some extent no-one knows what the role entails.  On the other hand, there is a wide-range of literature on evidence-based practice and in this post I will draw upon Barends, Rousseau and Briner's (2014) guide to evidence-based practice which identifies some of the misconceptions associated with such an approach.   In doing so I will try and provide advance warning of a number of pitfalls and misconceptions which can be avoided when undertaking the school research leads role.

Misconception 1: A research and evidence informed school is all about conducting in-house research

In the first instance a research and evidence informed school is in all likelihood not going to have the capacity and capability to undertake 'scientific' research based on the principles of disciplined inquiry.  At the first, a research and evidence informed school should seek to develop its' capacity as a critical consumer and user of research within a process of evidence-based decision-making.  Indeed, part of the process of evidence-based practice is subsequently assessing the impact of the decision which has been made, with in-house evaluation will becoming and integral part of the process.

Misconception 2 : Research is the same as evaluation

Fain (2005) cites Stuffbeam (2001) whow argues that the purpose of evaluation is to bring about improvement, whereas the purpose of research is to prove.  Now without wishing to be drawn into the 'science-wars' and whether anything can be 'proved' the fundamental point is sound.  Evaluation sits within the framework of evidence-based practice, suggesting that a judgement needs to be made as to whether a particular innovation/intervention has brought about improvement for a group of pupils, colleagues or the school as a whole.  On the other hand, research is seeking to provide generalisable knowledge for the 'population' as a whole.  However, that does not mean that the evaluation should not be a rigorous process, consistent with both disciplined inquiry and research.

Misconception 3: If you do not have a number of staff undertaking a masters or doctoral degree it is not possible to be a research-led school.

It is certainly desirable for those who are leading a school's research efforts to have experience at both masters and doctoral level.  That said, it does not require either a masters degree or doctorate to get better at asking well-formulated and answerable questions which can be informed by various sources of evidence.  Nor does it require 'masters' level education to understand and apply the principles of evidence-based practice.

Misconception 4: Good-quality EEF evidence gives you the answer to the problem.

Accessing high quality research evidence is certainly part of the process of engaging in research led or evidence-based decision-making.   However, that research evidence sits alongside other sources of evidence.  To paraphrase Dylan Wiliam - most things work somewhere, not everything works everywhere - the skill is to find what could work in your setting by taking into the account the particular needs of pupils, colleagues and the school

Misconception 5: Each school is unique, so the usefulness of research evidence is extremely limited.

Certainly each school will have its own context and challenge, and the school will no doubt will have to come up with its own evidence informed decisions about how to proceed when addressing a particular issue or challenge.  On the other hand, research can help evidence-based practitioners and school research leads become more 'intelligent problem-solvers' by being aware of what worked or do did not work elsewhere and what conditions maybe necessary but not sufficient for success.

Misconception 6: A research informed school will ignore the professional experience of teachers

In previous posts I have written that evidence-based practice embraces the practitioner's, and in this case the teacher's expertise.  The definition of evidence-based practice makes explicit the role of the teacher's expertise in making conscientious and well informed decisions and judgements.  The latest report from the EEF maybe important, but is no more important than the three other sources of evidence which are drawn upon, i.e.  teacher expertise, school data and the views of stakeholders, be it pupils, parents or others.

Misconception 7: Headteachers and teachers do not have the time to engage in research and evidence-based practice

Schools are time-pressured environments, that said evidence-based practice often involves taking a moment to 'step-outside' the current pressures being faced to reflect on the broader evidence available and make a more informed decision.  Furthermore, given the nature of the school-year, there is often a substantial period of time between discussion, decision and subsequent implementation.  This time could be used to ensure that various sources of evidence are drawn upon to answer well-formulated question, which get to the heart of the matter at hand.

To conclude:

There is no doubt that the increased interest in research in schools is to be both welcomed and encouraged.  However, given the importance of the work we are engaged in, occasionally it is necessary just to step back and reflect on the various directions we are going and clarify the terms we are using   Indeed, there is a real-risk that by focussing on research we take both colleagues and schools in the wrong direction by trying to conduct 'proveable' research.  Rather the role of the research lead should be one of helping colleagues become better consumers and users of research so that they engage in interventions which lead to improvement within their own practice and setting, and in doing so become better evidence-based practitioners.

References:
Barends, E., Rousseau, D. M., & Briner, R. B. 2014. Evidence-Based Management : The Basic Principles. Centre for Evidence Based Management (Ed.). Amsterdam.
Stufflebeam DL. Evaluation Models: A New Direction for Evaluation. New York, NY: Jossey-Bass; 2001.

Helping School Research Leads Ask Better Questions Part Three - The CIMO Format

In my last two posts I have looked at how the PICO format, which has been borrowed from evidence-based medicine, can help school research leads help colleagues ask well formulated and answerable questions.  In this post I will explore a similar approach,  
developed by Denyer & Tranfield (2009) for use in a social science or organisational context.  Denyer and Tranfield argue that well-crafted review questions need to take into both the organisational context and the relationship between an intervention and an outcome.  Adapting the work of (Pawson, 2006),  Denyer and Tranfield have developed a structured and contextual approach to developing an answerable question (CIMO) and which provides a better focus on both the context and the mechanism(s) by which change is brought about.

What is CIMO?

CIMO is an acronym for the the components of a well-formulated question for use in a social science or organisational context.

C — Context. Which individuals, relationships, institutional settings, or wider systems are being studied?

I — Intervention. The effects of what event, action, or activity are being studied?

M — Mechanisms. What are the mechanisms that explain the relationship between interventions and outcomes? Under what circumstances are these mechanisms activated or not activated?

O — Outcomes. What are the effects of the intervention? How will the outcomes be measured? What are the intended and unintended effects?

Denyer and Tranfield provide a worked example of a question framed with these components: 

“Under what conditions (C) does leadership style (I) influence the performance of project teams (O), and what mechanisms operate in the influence of leadership style (I) on project team performance (O)?” (Denyer and Tranfied, 2009 p 682)

Educational Examples

Using these elements it is now possible to frame answerable questions, several examples of which can be found below.

Under what circumstances does a further education college middle manager’s leadership style influence the academic performance of students, and what are the mechanisms of middle management leadership style which affect student performance (adapted from (Denyer & Tranfield, 2009)

Under what conditions does re-taking GCSE English provide an effective mechanism for developing 16 year old full-time further education students’ English skills, where those students previously achieved a grade D?  What are the processes associated with re-sitting GCSE English which affect English skills 

Is the use of flipped learning an effective mechanism for engaging  full-time 16 year old level one further education college students effective in reducing the risk of non-attendance, where there has previously been a history of non-attendance in school.  
What are the mechanisms of flipped learning which affect student attendance.

Under what circumstance are graded lesson observations effective in improving lecturers teaching where those teachers have previously been judged to be inadequate or requiring improvement, and what are the mechanisms of graded lesson observation which affect teacher performance.

So what are benefits of using CIMOs and phrasing questions in such a manner?. 

A number of benefits spring immediately to mind:

1. CIMO provides a framework for formulating problems in a structured manner, and the very process of developing the question promotes understanding of the issue at hand.
2. By formulating questions in this manner is that subsequently provides the basis for undertaking a systematic review and in particular provides guidance as to what literature to review and the data to be considered.
3. CIMOs provide  a basis for allowing researchers/bloggers and tweeters to attempt to agree the question to which they are trying to contribute.

In future posts I will consider how school research leads can help colleagues critically appraise the relevant literature and produce critically appraisal of various topics.

Denyer, D., & Tranfield, D. 2009. Producing a systematic review. In D. Buchanan, & A. Bryman (Eds.), The SAGE handbook of organizational research methods: 671-689. London: SAGE Publications Ltd.

Pawson, R. 2006. Evidence-based policy : A realist perspective. London: Sage Publications.

School Research Leads and Asking Better Questions - Part Two

In my previous post I adapted the work of Strauss et al (2011)  to help school-based research leads go about the task of devising well formulated questions.  In this post I intend to look further at the differing types of questions that can be asked.   But first a quick recap of the PICO format.   PICO is an acronym for the components of a well-formulated question and which are as follows:

P — Pupil or Problem. How would you describe the group of pupils or problem?
I — Intervention. What are you planning to do with your pupils?
C — Comparison. What is the alternative to the intervention (e.g.different intervention)?
O — Outcomes. What are the effects of the intervention?

PIOC and Different Question Types

Having created a well-formulated question, it is worth reflecting on the type of question-which can be created and which has been adapted from (Stillwell, Melnyk, & Williamson, 2010).

Pedagogical intervention - To determine which pedagogical intervention leads to the best outcome for pupils/outcomes

• For students requiring students requiring additional learning supported how does the provision of 1 to 1 support compared with group support affect retention rates in the first term?

Etiology - To determine greatest success/risk factors

• Are level 3 BTEC Extended Diploma students who have grade C or above in GCSE Mathematics compared with those students who do not, more likely to successfully complete their two year programme of study?

Diagnosis - To determine which test is more accurate in diagnosing learning  needs

• For post 16  students requiring support in the development of English skills,  are GCSE grades a better indicator of needs compared to specific on-line screening tool (eg BKSB)?

Prognosis or prediction To determine the course over time and likely complications of a particular pedagogical intervention

• Do weekly tutorials for students with poor records of attendance improve timely completion of coursework within three months of the initiation of the weekly tutorials?

Meaning To understand the meaning of an experience for a particular group of  students

• How do further educations students with grade D or below in GCSE English perceive re-siting GCSE English during the first year of post-16 education?

How to ask the most appropriate question?

Straus et al 2011 have suggested a series of filters which could be used to identify the most appropriate question to ask in a particular situation.  I have adapted the suggested filters so they can be easily transferred to an educational setting.

• Which question, if answered, will be most useful for our pupils/students/learners' well being - academic or personal?
• Which question will be most useful for subject leaders, heads of department in gaining a better understanding of the issues at hand?
• Which question will be most useful in helping to improve the department, school or college?
• Which question is most likely to re-occur and will need to be revisited in the future?
• Which question is most interesting to you as an evidence-based practitioner and contribute most to your personal professional development?

I'm sure that some colleagues are saying that they do not have sufficient time to formulate questions in such a structured manner.  On the other hand, as Strauss et al (2010) so clearly articulate - again amended for evidence-based educational practitioners - there are a number of clear benefits of such an approach:

1. Focussing our scarce professional development time on the needs of our learners.
2. Making it easier to communicate with colleagues.
3. As our knowledge grows we are role-modelling to our colleagues what it looks like to be a research informed practitioners

I hope you agree with me that by asking better questions we can expand our skills as evidence-based practitioners.  In future posts  I will continue to explore the challenge of asking better questions and will be drawing upon a number of differing perspectives.

References
Straus, S.E., Glasziou, P., Richardson, W. S. & Haynes, B.R. (2011)  Evidence Based Medicine : How to practice and teach it, (4th edition), Churchill Livingston.
Stillwell, S. B., Melnyk, B. M., & Williamson, K. M. (2010). Asking the Clinical Question : A key step in evidence based practice. American Journal of Nursing, 210(3).

The school research lead and asking better questions - part one

At the recent ResearchED Research Leads one-day conference one of the tasks that emerged for the role of the school-research lead is to help colleagues ask well-formulated and answerable questions. Indeed, there was some discussion as to whether this required input from a higher education institution.  Fortunately for both current and prospective research leads there is a wealth of material produced by the evidence-based medicine movement (Straus, S, Glasziou, P., Richardson, W.S.and Haynes, R.B (2011) Evidence-Based Medicine : How to practice and teaching it, 4th edition) which can help with the development of well-formulated and answerable questions, which I will now adapt for the use of school research leads.

Background and Foreground Questions

Straus et al state that in the first instance it is necessary to make the distinction between background and foreground questions, with background questions being made up of two parts:
* A question root (who, what, how, when, how ) with a verb
* An issue or matter of interest

So examples of educational background questions could be:

* How does homework improve student achievement?
* What are the benefits of e-learning?
* When is the best-time to give students diagnostic tests?
* Who is best placed to undertake performance reviews and appraisals?
* Where can you find examples of effective 'flipped' learning

Whereas a foreground question asks far more specific questions about a particular action, intervention or innovation, for example, does 24/7 access to iPads compared to the use of Chromebooks improve the timely completion of homework tasks. 

The PICO Format

Invariably foreground questions can be constructed using the PICO format developed for evidence-based medicine and which contains 4 basic elements which I have adapted for use in schools.

P —  Pupil or Problem. How would you describe the group of pupils or problem?
I  —  Intervention. What are you planning to do with your pupils?
C —  Comparison. What is the alternative to the intervention/action/innovations
O —  Outcomes. What are the effects of the intervention/action/intervention?

Two examples of foreground questions could be :

1. For pupils requiring additional learning support (P) how does the provision of 1 to 1 support (I) compared with group support (C) affect achievement rates.
2. For pupils aged 16 who failed to achieve at least at a grade C in GCSE English (P) and subsequently retake GCSE English (I) at the end of the academic year, how well do they achieve (O) compared to students who have been prepared and entered for iGCSE English (C)

Benefits for school-research leads
So what are the benefits for school research leads of using the PICO format, well for me three benefits come to mind.

1. PICO provides a mechanism for developing answerable questions, which should make it easier to pinpoint the relevant evidence necessary to answer the question. 
2. PICO provides a structure by which the staff who are being supported by research leads are able to ask better questions for themselves.
3. PICO should reduce the time-pressure on research leads as it will lead to far more empowered colleagues.

Given these provisional benefits of using the the PICO format,  in future posts I will look at extending the use of the PICO format to help ask a far greater range of well-formulated questions.